At Xmrit we are obsessed with variation, and how to make business decisions in the face of it. But why do we focus on it so much, when most people focus on averages?

To explain, let’s begin with a story from manufacturing; where much of the original work on variation in business started, including the XmR chart. This particular story comes from Introduction to Statistical Quality Control, 6th Edition by Douglas C. Montgomery (pages 6–7), but there are many similar ones I could have chosen.

Quality Means Less Variation - Case Study From The Automotive Industry

Many years ago an American car manufacturer was reviewing warranty claims and repair costs for its transmissions; my guess is it was GM during its partnership with Toyota at NUMMI, but I am not sure. To their surprise they discovered that transmissions made by their American supplier had much higher warranty claims and repair costs, compared to their Japanese supplier. This was despite all the American transmissions meeting the quality specification the manufacturer had set.

To understand why the performance was so different the car manufacturer began randomly testing the transmissions, across several physical dimensions known to be important to quality. After testing many different transmissions the car manufacturer produced comparison graphs like the one below.

Image adapted from Figure 1.2 in Introduction to Statistical Quality Control, 6th Edition by Douglas C. Montgomery, Page 7

The graph showed several surprising facts about the transmissions:

1. The average value for the transmissions were the same for the American and Japanese suppliers, centering on the target value set by the manufacturer.
2. Both the American and Japanese suppliers met the specification set by the manufacturer.
3. The difference lay in variability of the transmissions. The Japanese produced transmissions with much less variation than the Americans.

This lower variation meant the Japanese transmissions ran more smoothly, broke down less often, and customers reported them as being much quieter than the American versions. All this was true despite the both sets of transmissions meeting the “spec”.

This story illustrates a foundational truth in manufacturing:

Quality is a result of reducing the variation of your products, not the average value

This insight became the cornerstone of Statistical Process Control (SPC) and tools like XmR charts, which help monitor and reduce variation over time.

Why Variation Matters Outside of Manufacturing

If you don’t work in manufacturing you may be wondering how relevant this is to you. Your products don’t need tight tolerances to run more quietly, so is variation less important? But it turns out that variation determines quality just as much in the service and digital sectors as it does in manufacturing.

There are two reasons that variation determines quality in any industry:

1. Customers judge you by your worst day
2. High variation makes it harder to improve your product and business

Let’s have a look at those two ideas in detail.

Customers Judge You By Your Worst Day

If most of the time you give your customers a great experience, but sometimes you drop the ball and completely fail them, you can guess what the reviews are going to look like. Since the customers can’t rely on you, they have to plan for you to perform at your worst, limiting how much people will use or pay for your product.

Like with the transmission story we have another foundational truth about business:

Your customers will forget your best day, and judge your quality by its worst.

Let’s have a look at several examples where variability can kill, or make, a business outside the manufacturing sector:

• E-Commerce Delivery Times: McKinsey did an analysis of what US consumers wanted from E-commerce delivery and found that consumers value on-time delivery nearly twice as much as either 1-day or same-day delivery.
• Ride Hailing Cancellation: Research has shown that one of the biggest reasons customers will consider switching apps is because of an unexpected cancellation. By reducing the chance (variability) of a driver cancelling on you they are able to avoid having to spend money to win you back as a customer.
• AI Hallucinations: One of the most timely examples, as of May 2025, is the hallucination problem with AI. We have these incredible AI models which can write amazing poetry, solve complex math questions, and even talk us through a complex social situation - but we can’t trust them to work autonomously on many tasks because they regularly hallucinate - code for making stuff up.

Regardless of the industry, customers can only trust your product when it is reliable. As we can see with the above examples people are willing to pay for more trustworthy and reliable products, and are less likely to look for alternatives.

Variation Hides Progress

The second reason that variation is important is that a highly variable process makes it harder to know when you have made an improvement. Consider the two XmR charts below—both measure processes from January and February 2025, and both lock control limits using January data.

• The top chart shows a low-variation process, indicated by the tight process limits (6.75 to 9.15). A clear process shift appears in February, supported by exceptional variation signals.
• The bottom chart shows a high-variation process, indicated by the wide process limits (4.58 to 11.46). Even though the February average is lower than January there is no signal of exceptional variation.

But what I haven’t shown you is the average values for January and February are the same in both graphs!

Highly variable processes mean you take longer to detect changes, and struggle to detect small changes. This leaves you vulnerable to lower variability competitors, who can react more quickly to changes in their business and the market.

Thinking About Variation in Your Processes

Now that you know that reducing variation in your products and processes is the key to improving their quality, what should you be doing differently? There are two takeaways I recommend to people who want to practically use this knowledge in their business.

1. Look At Variation And Average When Assessing Process Improvements

When analysing process improvements you should encourage your employees and co-workers to review the changes in variation, as well as the average. It is easy to declare victory when you see the average value change, and forget to check if you have made a lower variation, more reliable, higher quality process.

The easiest way to do this is by plotting your metrics on an XmR chart, with free tools such as Xmrit. Using process limits, dividers, and the MR chart you are able to quantitatively assess the variation and the average of a process, before and after a change.

2. Remove Exceptional Variation First, Before The Average

If you have been asked to improve a poorly performing process with lots of exceptional variation it is easy to want to fix everything. But this is a mistake, you should focus on removing all the exceptional variation, before making any big changes to shift the average. Commoncog had a great summary of why this is true:

If a process displays both routine and exceptional variation, it is unpredictable. Hence: if the process is unpredictable, you need to first investigate and then remove exceptional variation. If the process is predictable, then you need to completely rethink the process. (A predictable process is already running the best it can, and the only way to change process behaviour is to fundamentally change the underlying process.)

Only once your process is predictable, within the limits of the existing process, should you start to rethink the process to shift the average. At that point having a less variable process will also allow you to more easily detect if those changes you did make were successful

Appendix

Common Misunderstandings About Quality and Variation

When I write that reducing variation is the key to high quality there are a couple of common misunderstandings that are important to clear up.

Misunderstanding 1: Quality Is Not Absolute

“If quality is just about variation does that mean McDonald’s, with its highly standardised processes, delivers better food than a fine dining restaurant?”

One of the most common misunderstandings about quality is that it’s an absolute standard, a fixed benchmark, that all businesses in an industry are striving to meet. This leads to confusion when we say that quality is about reducing variation, not just improving the average.

But this misses the point, Quality must be judged relative to the target and context.

McDonald’s is not competing with a fine dining restaurant, but instead with other quick service restaurants. Due to its different offering it has to balance delivering consistent meals across thousands of locations. A Big Mac in Tokyo should taste the same as one in New York. While a Michelin-starred restaurant has a completely different target: culinary creativity, seasonal ingredients, and deeply personalized service.

But even within the fine dining industry there is lots of process control, with the goal of providing highly standardised service to their customers. If you want to go deeper Commoncog has written extensively on the book Unreasonable Hospitality, which goes deep into the processes and tool kits used at the restaurant Eleven Madison Gardens.

Misunderstanding 2: High Output Variation Needs High Input Variation

“Industries that live on home‑run pay‑offs (pharma, VC, hit‑driven media, etc.) need wild, high‑variance processes to create those outlier outcome”

The second misunderstanding is to look at industries that have high variability outcomes, where a small number of outcomes lead to the majority of the results for a business, and think that requires them to have high variability processes.

It turns out this is the opposite of what happens! Companies in these industries have highly rigorous upstream processes, to avoid the organisation pursuing mistaken paths at high risk and high cost.

Let’s have a look at a couple of industry examples which demonstrate this point.

Pharmaceutical Companies

Pharmaceutical companies make the majority of their profits off a very small percentage of the drugs they enter for testing. The challenge is that the early testing phases are cheap, but the later ones become far more expensive and risky.

To make the drugs they put towards these late stage tests more likely to be successful pharmaceutical companies have rigorous and standardised vetting procedures:

• Standardised Stage‑Gate reviews that kill weak candidates early.
• Randomised, double‑blind protocols written to Good Clinical Practice (GCP) and audited by third‑party monitors.
• Good Manufacturing Practice (GMP) checklists that specify everything from room airflow to batch records—precisely so that any deviation is detected before it can poison a trial or trigger a recall.

Venture Capital Firms

Venture capital (VC) has similar high variability outcomes, with a small number of unicorns in their portfolios making the majority of the profit in most funds. In much the same way as pharmaceutical companies VC firms have implemented standardised de risking processes into their businesses:

• From YC’s SAFE agreement to NCVA templates, the VC industry has moved toward ever more standardised contracting, removing a significant area of variability in their agreements with startups.
• Seed rounds allow VCs to only put enough money in to get a startup to its next milestone. If a startup misses their goals it allows the VCs pull the plug early, targeting capital towards the next prospect.
• Extensive due diligence processes, such as A16Z’s Data Rooms, to reduce the chance they make a bad investment.