One of the things that we emphasise in the Metrics Masterclass is the idea that XmR charts come built-in with a specific approach to improvement.
The approach goes like this:
- If you see exceptional variation in a chart, the metric you’re looking at is unpredictable, and to improve it, you’ll need to investigate — and possibly remove! — the observed exceptional variation.
- If you see only routine variation in a chart, the metric you’re observing is predictable, and to improve it, you’ll need to rethink the entire process. This is because the process is already performing as well as it can; your goal for improvement is to change things until you see tighter variation, or a shift in the entire range of variation upwards (or downwards, depending on which is considered better).
This two-prong recommendation seems simple. As I write this I can see you nodding your head (or rolling your eyes). But I’ve found that in practice, this set of recommendations produces something quite remarkable: it means that whenever you put a metric on an XmR chart, you know exactly what to do. There’s no longer any question about “so what?” or “what should I do about this?” — which so often happens when you’re looking at a wiggling chart in the context of a dashboard.
If you have an XmR chart, you automatically know how to improve.
In Manufacturing, Exceptional Variation is Bad …
XmR charts were originally developed for and used in manufacturing. The goal in manufacturing is in some ways simple: make parts with consistently tight tolerances. In this domain, exceptional variation is bad — we don’t want gears with ratios that are ever so slightly off, nor do we want furniture that just barely fits together.
Consistency of fit and finish is so important, in fact, that you will often find videos of car reviewers examining inconsistent seams and slightly malformed joints as proof of bad build quality. (Tesla, in particular, comes to mind here — there are a gajillion videos of badly built Tesla cars on YouTube, and I watch a dozen of them every couple of years, for fun).
As a result, it’s not very surprising to see all the books on manufacturing — which means nearly all the literature on XmR charts — argue that you should remove exceptional variation whenever you spot it. They will say things like: “if exceptional variation is present, something unpredictable is going on, which means you can’t improve your process. If you attempt to improve things without removing that exceptional variation, random things will occur to stymie your efforts. Exceptional variation is bad! So investigate and remove it first.”
One famous Six Sigma saying is “customers don’t feel averages, they feel the variation.” The implication, of course, is that excessive variation in production is always bad. So: a) you want zero exceptional variation, and b) whatever variation you have, you want it to be tight.
Outside of Manufacturing, Exceptional Variation Can Be a Gift
We know, however, that XmR charts aren’t just useful in manufacturing. In 1939, statistician and XmR charts pioneer W. Edwards Deming used Process Behaviour Charts to measure and then improve the performance of census operators at the US Census Bureau. To be precise, what he did was that he sampled 5% of the operators’ work to determine which operators were good, and which operators’ work fell outside the limits. The underperforming operators were pulled out and retrained. Deming also identified stellar operators who were wasted as census inspectors, and pulled them back to become punch card operators instead (whilst the underperformers were being retrained). This was the first time such charts had been used outside of manufacturing … and it worked, of course. By the time he was done Deming saved the Bureau a sweet $263,000 annually, around $5 million today (source).
More importantly, we know that XmR charts work just as well in broader business contexts, such as with measuring newsletter subscriber growth, or instrumenting blog visitors, or measuring the impact of ad campaigns. We’ve applied XmR charts to our respective business contexts (Sam works at a big tech company; I, Cedric, run Commoncog), and we’ve seen their effectiveness first hand.
And what we’ve found is this: in manufacturing, exceptional variation is nearly always a bad thing. Outside of manufacturing, however, exceptional variation can often be a gift.
What do we mean by this?
When you’re running a website or a newsletter (as many B2B software companies do), it is not uncommon to see positive exceptional variation on a month-to-month basis. Here is a chart of newsletter subscribers from Commoncog, for instance:
What happened here?
The first point of exceptional variation — with 116 new newsletter subs — was the result of landing on the front page of Hacker News (a major tech industry news aggregator). This led to a spike across a number of different metrics, including total unique visitors and ‘In-Depth Readers’ (number of visitors who read more than one article).
In the second, higher peak — spread across two weeks — Commoncog was linked to from a major personal finance newsletter in India. Interestingly, unlike with the Hacker News spike, we did not see equivalently high peaks across other website metrics. It was almost as if visitors hit the linked article, went to the Commoncog front page (where there is an explainer for the website and a newsletter form), and then signed up.
How would you use this information?
To give you some context, we had been trying to shift the process behaviour of the new newsletter subs metric for a few months now. Earlier, when we were new to XmR charts and lacked knowledge about our own business, we tried an experiment where we posted more on social media. We also tried an experiment where we redesigned the newsletter forms. Given Commoncog’s niche status and target audience (executives and investors), a great growth rate would be on the order of ~100 new newsletter subs a week. Very little of what we’ve tried has resulted in exceptional variation in our new newsletter subs metric. But here are two events where we’ve observed an exceptional spike in exactly the metric we want to influence.
In order to turn these observations into knowledge, we have to come up with a hypothesis for why these events resulted in exceptional variation. A hypothesis is the beginning of predictive knowledge.
It will take up too much context to explain all our hypotheses — including which of those hypotheses we chose to test. In the interest of brevity, here are some hypotheses that you may reasonably come up with, when presented with this information:
- Hacker News virality is mostly luck-based, but are a function of submission consistency and luck. We should increase submission consistency for non-paywalled posts.
- Newsletters are a better medium to target for new newsletter subs. (But what types of newsletters? And how do you get them to link to you? How do you set this experiment up?)
- In the past, heavy posting to Twitter and LinkedIn has not resulted in exceptional variation in this metric, nor has it resulted in a process behaviour change in the metric. We should de-emphasise the importance of social media posting for this output metric and experiment with these other avenues instead.
Note that in this story, we are not guaranteed to find a repeatable and scaleable lever for increased newsletter subscribers. What we do have, however, are two clues. In business, it is actually quite difficult to find things that reliably impact the metrics you care about. Having clues helps narrow the range of possible things you may experiment with; it prevents you from throwing everything at the wall and hoping that something sticks.
I should note that we have found some things that work, but this is a hard-won secret and I do not wish to write about it publicly. But I am 100% sure that you would be able to find levers that work, if you were given the above information, and you pursued the implied experiments with vigour.
The point I am making here is this: outside of manufacturing, every point of exceptional variation is an opportunity. I like to think of it as a gift. In the manufacturing textbooks, exceptional variation is seen as a bad thing. But in business more broadly, exceptional variation can be delightful. Either way, it is an invitation for improvement:
- If it’s bad exceptional variation, you have an opportunity to prevent the bad thing from happening in the future.
- But if it’s good exceptional variation, it is a clue as to what might actually affect your metric.
Sometimes good exceptional variation is the result of a deliberate improvement attempt. But — as we’ve just shown you — it can also be delightfully unexpected. We never expected a large finance newsletter linking to us. Nor did we predict we would end up on the front page of Hacker News. But the net result is the same: we have more opportunities to figure out which are good things or bad things, so that we may avoid the bad things, and do more of the good things.
It’s as simple as that.