Statistics and Quality: An Inverse Relationship?
“There are lies, damned lies, and statistics.”
This insightful phrase, popularized by Mark Twain, succinctly highlights the misleading nature of statistics. Statistics, or metrics, are often used to measure performance. If used correctly, they can be an enormous aid in continuous improvement, by providing the data required to evaluate root cause and develop corrective actions. If used incorrectly, they can waste significant resources by acting as a red herring in the problem-solving process. In some cases, they may show that a problem has been solved when in reality it persists or has been made worse. Twain made this observation over 100 years ago, but the nature of statistics and the potential for their inverse relationship with Quality remains the same a century later.
Quality guru and author of several books on organizational quality, Philip Crosby, makes a point while discussing his “Zero Defect” mentality in Quality is Free that illustrates how, if used incorrectly, statistics can be used to justify an otherwise unacceptably low level of Quality. He points out that “we [don’t] expect hospital nurses to drop a certain percentage of all newborn babies.” Imagine if a hospital used statistics to measure a nurse’s performance in this area. Even if the quality target set for hospital nurses holding newborn babies was 99.9%, that still means that it would be acceptable to drop 1 in every 1000 babies without considering any change in baby-holding procedure or any corrective actions, like retraining on how to hold a baby.
Even in quality engineering, statistics can be used to justify low levels of Quality if used incorrectly. There is a statistical tool used by quality engineers called Acceptable Quality Levels (AQL) that assists in identifying appropriate sampling plans for inspection. The inspector selects an AQL Level, typically based on the criticality of a part or dimension. It is then cross-referenced against the total quantity of the material being measured using a table developed through statistical methods. The output is how many samples are required to be measured to accept the material. A common misunderstanding is that, since the material is accepted, it is all conforming to specifications. The AQL Level selected actually represents the percentage of nonconforming material allowed. For example, if an AQL Level of 4.0 is selected, it is being said that it is acceptable for up to 4% of the accepted material to be nonconforming. This is typically not communicated downstream or to the customer. All that is communicated is that the material passed inspection. The assumption by the customer or by whoever receives the material downstream is that the material is conforming. This assumption is often reinforced by a certificate of conformance, but all that is being certified is that the manufacturer has confidence that at least 96% of the material is conforming.
The use of statistics has certainly contributed to an increase in productivity and process efficiency. They can be incredibly useful if they are understood and used correctly. A tool like AQL can be used to significantly reduce inspection times for components with the appropriate downstream risk mitigation, but depending on where it is implemented, it can also be used to justify releasing material with a 4% defect rate to the market, and as Crosby points out, they can sometimes be used to justify the dropping of newborn babies in the hospital. A great example of the potential for statistics to have an inverse relationship with Quality is the statistic used to measure inflation in the United States.
On Inflation
Inflation, at least in the United States, is determined and communicated to the public using a measurement called the Consumer Price Index (CPI). According to the U.S. Bureau of Labor Statistics, “CPI is a measure of the average change over time in the prices paid by urban consumers for a market basket of consumer goods and services.”
One market basket that the CPI uses to measure inflation is “food and beverages” and as the name implies, it is a basket of what consumers are spending on food (and beverages). This is the average price of commonly purchased food items, not, for example, what the price of steak is. To show how this works out, let’s go through an example of the decisions an individual consumer makes in the face of inflationary pressures.
For simplicity’s sake, let’s say that a family consumes 5lbs of meat a week. In September 2021, they purchase 3lbs of steak for $36 and 2lbs of chicken for $8, for a total of $44. In September 2022, they purchase 2lbs of steak for $32 and 3lbs of chicken for $15, for a total of $47. Using the CPI, the published rate of inflation for September is 6.8%. Steak, during that time, saw real inflation of 33%, going from $12/lb to $16/lb; and chicken saw real inflation of 25%, going from $4/lb to $5/lb. If the consumer did not adjust their spending habits and maintained their level of Quality, purchasing 3lbs of steak and 2lbs of chicken for $58, real inflation would be 31.8%. That is a big difference from the 6.8% claimed by the CPI.
In U.S. Monetary Policy and the Quality of Everything, I provide an example that is helpful to understand how a degradation of Quality can be concealed by the way inflation is measured. This example illustrates the decisions an organization makes in the face of inflationary pressures before the choice even gets to the consumer. In the example, the car manufacturer decided to make a change to the material of the body of the car from metal to plastic. This Quality-reducing change allows them to maintain the same manufacturer’s suggested retail price (MSRP) for an equivalent model in the next year. So, say in September 2021, if a family needs to purchase a car, a new Honda Civic might cost them $25,000. In September 2022, a new Honda Civic would still cost them $25,000, however the body of the car would be made of plastic instead of metal. The CPI attempts to account for this change with what is called a “cost-based quality adjustment,” which evaluates changes in the specifications of a new vehicle model from year to year. If the adjustment for this change is $1,000, the CPI would show 4.2% inflation for the year, which is certainly more accurate than the 0% inflation that would otherwise be represented. The new Honda Civic, however, would experience significant long-term Quality effects, such as increased rate of depreciation and lower resale value, compared to the 2021 model. Even with a good faith attempt to improve accuracy, a statistic like CPI can be misleading.
Measuring inflation this way assumes a degradation in Quality, taken on by organizations, consumers, or both. When the Federal Reserve decides to increase interest rates as a method of decreasing inflation, they are targeting the number provided to them by the U.S Bureau of Labor Statistics. Their goal is typically to return inflation, as determined by the CPI, to between 1-2%. When this is accomplished, the Federal Reserve views the problem as solved, but the corrective actions taken are ineffective, or at a minimum insufficient, because the Quality concessions made by consumers and organizations often remain unaddressed.
The CPI is a topical example of misleading statistics that is relevant due to the impact inflation is currently having on every individual and organization, but misleading statistics are used every day in all arenas of life. Statistics exist to provide the information required to identify issues and put in place corrective actions to address them, in other words, to increase Quality. Their ability to achieve this goal, however, depends heavily on how they are collected, presented, and interpreted. As I discussed in Truth, Quality, and Bullshit, I believe truth to be synonymous with Quality. Statistics, as alluded to by Twain in his observation over 100 years ago, are often used, whether intentionally or unintentionally, to obfuscate the truth. When decision makers are blind to the truth because the data used to inform their decisions are generated by misleading statistics, the inevitable result will be the degradation of Quality. Regardless of your role in society, everybody is a decision maker. Whether you are making decisions for your household, your organization, or your community, I recommend scrutinizing the statistics used to inform your decision making and examining their relationship to Quality.