In Cedric’s last article he raised the question how would you know? As he wrote there is no single answer to this question, but the key thing is to have decided how you would know before you make a change. At Xmrit we often recommend the simple approach of deciding which input and output metrics you expect to shift as a result of your change, and plotting these metrics on an XmR chart.

Putting in the effort to check if your changes have had the effect you expected is hard. It requires you to think through what you expect to happen before you make a change. Then also have the discipline to check if what you expected to happen actually happened. Doing this well is hard, and many businesses fail to properly check the impact of their changes.

But many other businesses make a worse error, the error of superstition.

What Does Superstition in Business Look Like?

The idea of superstition in business might seem surprising. We usually associate superstition with supernatural beliefs—like ghosts or witches—not with how we manage our businesses. But superstition can just as easily describe any unexamined or untested belief, especially when it shapes how we make decisions.

In everyday life, supernatural superstitions (like a fear of ghosts) rarely cause significant harm. But in business, untested beliefs about how your company operates can lead to missed opportunities or costly mistakes. They can lead you to make decisions that at best are less effective than you hoped. At their worst they can lose you money or put your business in jeopardy.

But what does having superstition in business look like? Let’s have a look at a couple of examples, one from Uber, a large data driven company, and one from Commoncog, a slightly smaller but still data driven company.

Uber’s $120M Mistake: Digital Advertising Superstition

In 2017 Uber discovered that adverts it had placed to promote Uber were appearing on the controversial media site Breitbart. These adverts generated significant negative publicity for Uber, and they required their ad network providers to block all advertising to Breitbart. However, some advertising networks failed to fully block Breitbart adverts, resulting in continued negative publicity for Uber.

To fully eliminate any ongoing Breitbart adverts appearing, Uber turned off all payments to networks that had failed to block the adverts, ~10% of Uber’s advertising spend. This was expected to have a significant impact on driver and user signups, as these networks were reporting they were responsible for large numbers of Uber app signups and rides.

But when Uber turned off the advertising… nothing happened.

Signups stayed the same after the spending had been turned off. Great you may think, Uber just saved 10% of its advertising spend. But how was that possible? On further investigation it turned out that the advertising networks were defrauding Uber, falsely attributing signups to their ad network that were already going to happen. Uber turned off another ⅔ of their annual spend ($100M) and still saw no change in their signups!

Uber’s belief about how online advertising worked for their business was nothing more than superstition.

This superstition took hold because Uber didn’t test the true impact of their spend. It is easy to imagine being in the marketing department and thinking “everyone does online adverts and I see a report from our partners with great numbers, it must work”. But Uber never ran experiments to truly validate the impact of these online advertising campaigns. It was only after the fraud was discovered that Uber started experimenting with the efficacy of their advertising spend. As a result of these experiments Uber eventually ended up keeping only $30M of their original $150M budget.

Uber by this point was eight years old, and had thousands of highly paid and highly analytical employees - yet they still failed to properly experiment and validate their beliefs. Don’t fall into the trap of thinking superstition could not happen to you as well.

Now lets have a look at a case of superstition closer to home, Commoncog’s superstition about how social media works for promoting signups to its newsletter.

Commoncog: The Myth of Social Media as a Simple Front Door

At Commoncog, we fell into the trap of superstition with our belief in the power of social media. For a long time, we assumed that social media was a critical driver of newsletter signups. Our mental model was simple: more page views from social media would lead to more subscribers.

But when we started using a Weekly Business Review (WBR) process to closely monitor our visitor and subscriber metrics, the reality didn’t match our expectations. Even when posts from Commoncog went viral on Twitter and LinkedIn, there was no measurable impact on newsletter signups. We initially thought there could be a lag effect, but despite continual monitoring we couldn’t identify any meaningful effect from social media views on signups.

This finding shocked us— as we had always seen social media as the “front door” to Commoncog.

Of course, it’s possible the effect of social media on Commoncog is subtle, but still important. For instance, it could contribute to long-term brand awareness or indirect signups. But it doesn’t operate in the straightforward way we believed it did. In this case, we realised we had been superstitious about the role social media played in our business.

So far we have decided against running more detailed experiments on the impact of social media, as we don’t expect high returns and can only run a limited number of experiments as a small team. Instead the team has dedicated time to investigating other channels for signups, ones with better evidence for their effectiveness.

Why Are Businesses Superstitious?

It is surprising that people and businesses are superstitious, especially when there is money to be made by understanding your business better! My theory is that superstition is the natural default way that humans think. The effort and discipline to check your beliefs is hard, and human brains are constantly fighting to find ways to reduce the amount of work they need to do.

The challenge that businesses have is that they can be enormously complicated systems. Being clear about what evidence we have for our beliefs, and which beliefs are contributing to poor performance can be difficult. So difficult that we look for shortcuts to try and reduce the amount of thinking and disciplined follow up we have to do.

Many of the shortcuts we find are by looking at other people or companies that are successful, and copying what they are doing:

• A big successful company in your industry is doing something, therefore not only must it be right but it will work for my business as well.
• A reputable consultancy or business thinker recommends an approach or framework, therefore you should apply it to your business.
• A friend in a similar business has found success from advertising using a new channel, therefore it must also work for you.

In many cases copying best practice can have good results. You can think of best practice as sensible defaults for your business. But, if you never do the work to validate if what you are copying actually works your business is being built out of bricks of superstition.

Building Contextual Knowledge - Breaking Out of Superstition

To break out of superstition requires you to be constantly testing your beliefs against reality. Repeatedly testing your beliefs helps you to build true knowledge of how your business works, which then allows you to take control of it. The challenge is that this requires effort and discipline to plan your experiments, and then follow up on the results to validate they had the impact you expected.

Experimenting to build knowledge is not only hard for you as an individual, but even harder to scale repeatably across an organisation. However, if you are able to do it you can build an operational competitive advantage against your competitors [1].

Importantly the knowledge you build is often highly specific to your business and its context. A fun example of how contextual knowledge can be is comparing the impact of social media on Xmrit vs Commoncog. Although we haven’t tested this fully, initial indications are that social media posts for Xmrit have a significant direct impact on signups and purchases - the opposite behaviour that we see for Commoncog.

There are many potential reasons for the different behaviour, such as Xmrit being much newer and having low brand awareness, or Xmrit being a data focused product not a general business product like Commoncog. Regardless of the reason, it was important that we did not just blindly copy our learnings from Commoncog over to Xmrit, despite the similarities between the businesses.

In the Metrics Masterclass we have several videos on how you can build a culture of experimentation, and start to take control of your business. We discuss topics such as the Deming PDSA Loop, building a culture of experimentation inside your organisation which we describe as a “Process Control” culture, and the three questions of continuous improvement you should ask yourself before starting any change.

Building a business culture that has the habit of continually asking “how would you know?” is hard, but as you have seen with Uber and Commoncog it is a worthwhile goal to aim for.

[1] One of my weakly held theories is that the concept of “Process Power” from the book 7 Powers by Hamilton Helmer derives from a culture of experimentation. In the Metrics Masterclass we talk about a culture of experimentation as a “Process Control” culture.