5 Things Ruining A/B Testing

August 10, 2019

A/B testing is fun. It’s exciting to launch an experiment. We are A/B testing different recipes – cooking to see what tastes right. The anticipation of the results is apple pie from the oven.

But something has happened over the years. A/B testing has matured – into an old fart. We were given a new technology to explore. But any time we want to try a pinch of salt or an extra apple, someone is slapping our creative hand to bake.

And our creativity comes from the output of A/B tests launched. For example, the more we launch the more we can refine our approach. Each iteration brings our recipes closer to an experience our end users will love.

Below are five archaic ideas. Avoid them. They will make you slow to launch your A/B tests.

1: OVER OPERATIONALIZE THE A/B TESTING PROCESS

Creating a hypothesis is a five minute process. A single A/B test doesn’t require 585 pages of Martha Stewart. For example, a few sentences and you are done. Anything more, you are overthinking it and wasting time.

2: CROWDSOURCING A/B TESTING IDEAS

Crowdsourcing your A/B testing ideas is welcoming constant distractions. That isn’t work. For example, two people make food. The chef, who provides direction. The line cook to execute. This is how A/B tests get deployed quickly. Crowdsourcing, spoils the broth.

3: PREDICTING RESULTS, THE CHARADE

Trying to guess or predict A/B testing results is a worthless exercise. Time wasted. It’s not useful to anyone. For example, even if you are right or wrong – who cares? No one is impressed by your luck to draw a number. Stop telling me how good it will taste and bake that pie. When you find something that wins double down on the results found.

4: CREATING A/B PRIORITIZATION TESTING ROADMAP

Have a vision, have a strategy, don’t have a long list of ideas. Creating an A/B prioritization roadmap is a facade. For example, no one is impressed by what pie you might bake. But pulling a fresh pie from an oven will get your boss, the neighbor, and cat running to your desk. Optics are different from five A/B tests planned vs five A/B tests launched. A giant list of A/B tests doesn’t help launch A/B tests – it only draws attention to what you have been doing all day.

5: BIG CHANGE, NOT REQUIRED

Don’t boil the ocean. Simple changes will yield great results. For example, your first move, don’t redesign anything. Change an image, move a button, update a headline. I created 15m in net new revenue with two words. And ice cream is 3 ingredients, and 5mins to make. Keep your recipes simple, and move fast.