Here’s a question that we get a fair amount: how often should you measure a process? And how often should you subject that metric to an XmR chart?

All told, this is a fairly basic question — one that you’ll ask early on in your data-driven journey. And there are really only three important things to think about.

How Quickly Does The Process Change?

The first and most obvious question to ask is “how quickly does the process change?” followed by “Don’t measure faster than the process can change!”

An example is in order. Let’s say that you’re working at a factory, and you are monitoring a chemical boiler. You want the temperature of this boiler to stay within a certain tight range. With modern sensors, you can measure the temperature of the boiler more than a hundred times a second. You can pipe this data to a data warehouse, and draw fancy charts around it, and stream the data over the internet to a remote dashboard so your company’s executives can check in on the performance of said boiler (and then stand up in a fancy tech conference and preach Internet of Things™ and Industrial Revolution 4.0™ with a straight face.)

But this is useless. Unless something goes catastrophically wrong, the temperature of your chemical boiler will not change that quickly. For most production problems that you will encounter, problems with your boiler will cause the temperature to drift slowly — which means that taking, say, the average of three measurements any time in the final 10 minutes of each hour should be sufficient to catch exceptional variation in the boiler.

This is important! This principle — “never measure faster than the rate of change of your process” — is actually the quintessential consideration for this question. In his 1993 book Understanding Variation, statistician and XmR chart expert Donald Wheeler explains that if you “measure at a grain that is higher than the rate of change of the process” you risk increasing the false alarm rate. This makes intuitive sense: for most real world processes, XmR charts have around a 3% false alarm rate. Increasing the number of observations to 100s of readings a second would invariably increase the odds of a false alarm.

As a result, it is worth asking yourself: “how quickly will a change show up in a process once I’ve improved it?”

1. If you are changing a sales process — say, via sales training — it probably makes sense to measure on a weekly grain. Changes from sales training do not show up so quickly because it takes some time for behaviour change to happen. As a result, measuring on a weekly cadence is ok: it takes about that long for changes to flow through to sales metrics.
2. If you are changing your website — say to test a hypothesis that a certain Call To Action section would convert better — you should measure at the grain of a day. (This is not a good example, as it’s actually possible to A/B test this. But let’s pretend that you’re too lazy to set up a proper A/B test, or you don’t have the tooling set up for this).

We’ve found that this question is quite easy to answer. This leads us to …

How Expensive Is It To Measure?

The next consideration is common sense. We are used to thinking in terms of digital instrumentation: Google Analytics tracking on your website, say, or in-app measurement. Digital tracking like this is cheap.

But there are many more metrics and processes in business that demand some form of manual measurement. For instance, let’s say that you’re rolling out new sales training. You don’t want to wait to see if your new sales training has made a difference to sales outcomes — this could take weeks. No, as we’ve discussed in a previous blog post, you’ll also want to measure the rate of sales pitch adoption.

In that essay, we gave the following ‘operational definition’ for one such metric:

1. Criterion: Sales Pitch Adoption
2. Procedure: Some 80% of sales calls are done virtually over Zoom (the rest are in-person), and are recorded using sales recording software. Of those recorded calls, five recordings are randomly picked every Wednesday and are evaluated as pass or fail by the sales manager for the region. Sales Pitch Adoption will be the % of successful calls across all the sampled calls across the entire sales org.
3. Decision Rule: The sales manager will pass or fail the recording based on a simple scoring rubric: a) that the sales call proceeds in the order: pitch then questions then demo, b) that the question segment of the call is no longer than 10 minutes, and c) that the salesperson successfully establishes a next step for the prospect at the end of the call.

Notice that such a metric requires you to get buy-in from sales managers! Also notice that there’s some manual work involved: your sales managers must be willing to enter data into a form every week or so.

This is costly. It is costly in terms of time, and costly in terms of political will (to establish the habit, you will need to follow up very consistently for a month or so). Hopefully the potential return is higher than the cost, but the cost is a consideration nevertheless.

There are other metrics that can be even more costly. For instance, in a podcast interview with me, Colin Bryar described a metric that they tracked in Amazon: ‘Inventory Picking Accuracy’. Amazon’s staff had noticed that if inventory is not where the software says it should be in the warehouse, then this inaccuracy is likely to lead to Amazon missing the promised delivery window. This was bad, of course: it meant that Amazon was delivering a subpar customer experience.

So they came up with a new metric. The following Operational Definition is a bit stylised, but we’re including this here for demonstration purposes:

1. Criterion: Inventory Picking Accuracy
2. Procedure: Every day at 7am, a Fulfilment Centre (Amazon’s name for ‘warehouse’) worker will open the Inventory App and start a ‘Random Stock Check’. The software will randomly display 15 items. The worker will walk around the FC floor to check if each item is where the software says it is.
3. Decision Rule: If the item is where the software says it is, the worker will tap the ‘yes’ button on the app. If not, the worker will tap ‘no’. Inventory picking accuracy is the percentage of ‘yes’s across all readings for that FC for that day’.

But notice: what are the costs of this? Should we measure twice a day — say once at 7am and once at 5pm? Should we tie measurement to the start of each shift so that behaviour adoption becomes easier? Should we check 10 items, or 15 items? Remember that checking more items takes more time, and therefore costs us more because it ties up a worker for longer. But checking too few items means that we don’t have a representative sample.

Each of these considerations are things that you have to think about when initiating a new count.

How Quickly Do You Want To Improve?

This final consideration was articulated to us by Luca Dellanna, an author and consultant based out of Italy.

Dellanna notes that how often we measure often determines how quickly we can improve.

• If you measure once a month, you only have 12 opportunities to improve a process in a given year.
• If you measure once a week, you have 50 opportunities to improve a process in a given year.
• If you measure every day, you have 260 opportunities to improve a process in a given year! (Assuming 260 working days — which might not be the case for certain vocations)
• And if you measure every hour, you have the opportunity to … well, you get the idea.

Dellanna continues:

• If you measure every day, a process failure showing up on your XmR chart means that the process was broken for at least 12 hours out of the past 24 hours.
• If you measure every week, a process failure showing up on your XmR chart means that the process was broken for at least three out of the previous five days.
• And if you measure only once a month, well … you get the idea here, don’t you?

The more often you measure, the shorter a process failure would have to run before you detect it.

In addition, how quickly you see a signal is also constrained by the minimum number of data points you’ll need to be confident of a process change. In many cases, you must wait for at least six new data points before you can conclude anything about a process change. This is covered elsewhere on this site.

There is one final point to make here. In many cases, “how often you measure” is a nicer way to say “how often you check your XmR chart. Measurement alone does not result in improvement. It is worth thinking through how you might ensure that a human with decision rights over the process checks at the right frequency.

Conclusion

When you are attempting to measure a process, ask yourself:

1. How quickly can this process change? And then make sure you measure no faster than the rate of change.
2. How costly is this measurement? Some measurements — especially digital ones — can be cheap. But in many business contexts, measuring something costs time, money, or political will.
3. How quickly do you want to improve? Note that the correct answer here might not be “as quickly as possible.” It really depends on your priorities, plus the two considerations above.

And there you have it! How often should you measure? Ask these three questions, and you’ll know your answer.

Special thanks to Luca Dellanna. We highly recommend his book Best Practices for Operational Excellence; Dellanna helps companies increase revenue through better people- and risk-management.