Statistical consultant, business philosopher (and process behaviour chart OG) W. Edwards Deming’s favourite question was “how would you know?”
As in: “how would you know that the thing you’re suggesting would work … or has worked?”
You can imagine the sort of contexts in which he would ask such a question: directed at a hapless manager, who is wavering a bit in the 100-watt stare of a tall, very grumpy old man.
Deming’s favourite question is a very good question to ask, as it turns out. He asked this of factory foremen and managers, executives and leaders of industrial companies in the post-war period. But the question is just as valid today.
A stylised example:
“Why do you want to move that machinery to a new building?”
“Because it would allow us to stack up more partially-assembled body kits in the main building.”
“And why would you want to do that?”
“Because then we can increase the throughput of our production line.”
“How would you know?”
“How would I know what?”
“That you’ve succeeded in increasing the throughput of your production line?”
A silence. And then: “Well, I’ll know through my experience!”
And then Deming would roar: “And with that, he has revealed himself!”
That is: he would have revealed himself as a man of superstition, not knowledge.
We Are No Better
It’s important to note that we all do the same thing, even if we are far removed from 70s-era industrial companies.
Why are we switching to agile? Because we want to increase developer productivity. But how would you know?
Why are we increasing our spend on Facebook ads? Because we need to spend on Facebook in order to attract new customers. But is it the best use of marginal acquisition dollars? How would you know?
Why are we moving to continuous deployment? Because there’s a study (and book) out there that shows that the best performing teams use DevOps best practices. Yes — but you are one company, not a study population. Even doctors do trial and error for medication that’s been proven to work in a randomised controlled trial. Drugs may work for 99% of the population, and fail to work for you. How would you know that these practices have given you the results you desire?
This seems like a tad overwrought. Surely, in that last example, it isn’t necessary to be so rigorous? Aren’t ‘best practices’ good for something? Why did Deming ask this question so often?
The answer is simple: if you know what truly works for your company and context, you will begin to build better intuitions for other things that might also work. You will begin to suggest creative new solutions to your business problems, bolstered by the confidence that your knowledge is real. And since every company is somewhat unique, it is important that you gain actual knowledge for what works for you.
More Than One Good Answer
You might think that we would argue that XmR charts are the best answer to “how would you know?” After all, we built a free, open source tool for XmR charts.
But you would be mistaken.
There are, in truth, multiple ways to answer this question. For instance:
- Running an A/B test is the gold standard answer to “how would you know?” — though running A/B tests demand some statistical sophistication, and may be expensive to run depending on your organisational setup. (For instance: do you have the necessarily tooling? Are the costs justified for the expected return?)
- Customer interviews are also a valid way to get feedback — though this may depend on the nature of the interview, your ability to run such interviews skilfully, and how long it takes before customers can give you good feedback. After all, some product features or process changes may take months before you can get good qualitative data.
- Finally, you can use your intuition — though this is the least acceptable answer, and one that Deming would likely chew you out on. But it is valid if there is no data that you can look at to help you decide, and it is relevant if you have a good track record of getting things right. (Note: most of our judgments aren’t that good — that’s why we use data to augment and check our intuitions). But we are not saying don’t use your intuition — we are saying to use it carefully, and to never default to your intuition as a first port of call. I once asked Colin Bryar why the S-team trusted Jeff Bezos’s suggestion for a particularly risky new program. After explaining that Amazon’s leadership had exhausted all possible analysis for the decision, he said “well, Jeff had a history of being right.” That answer has always stuck with me. I have since asked myself: “how can I ensure that I get things right more than I get things wrong?”
With all that said, it is true that Deming taught the process behaviour chart as the default answer to this question.
If you don’t have a good alternative, it’s very easy to place some metric on an XmR chart (such as with Xmrit’s free tool) and then check to see if your change has worked.
But the variety of these of answers obscure the importance of the question. It is, to put it simply, very useful to ask “how would you know?” whenever you are embarking on a new course of action.
At Commoncog and with Xmrit, we’ve found that it has remarkably easy to turn this into a habit.
Turn the question into a habit, and you should find that you’ll improve faster. And you should also know that asking the question repeatedly makes you part of a long line of operators who have become good in the exact same way. As Charles Koch — another Deming acolyte — liked to say: true knowledge should lead to effective action.
Knowledge, after all, is what separates the great operators from the merely good.