I was having a conversation with a colleague about the best way to assist a customer with their 2012 planning, and it brought up the third bias I deal with on a common basis, Hyperbolic Discounting or “the tendency for people to have a stronger preference for more immediate payoffs relative to later payoffs, where the tendency increases the closer to the present both payoffs are.” Besides being the basis for the entire credit card industry, in optimization this mostly comes out in terms of better versus best testing.

When groups decide on a roadmap, or the list of tests that they mistakenly refer to as a roadmap, they will often prioritize them by what is most on top of people’s minds, or what their boss wants. You have all been sitting around and think that changing the button to Buy instead of Buy Now, or you really want to change the background on your promotional images on your front door, or the copy on your landing page, you think it needs to say X… All of this leads to the want to test your theory and prove yourself right. You shut down, ignore discipline, and try to see if that idea is better than the other. The entire concept of hypothesis testing leads to this failure of human cognition. It works if the only goal is to prove a single point, but it is really inefficient if the goal is measure the relative value of an action. We forget that we are dealing with more then the day to day issues in front of us, and we try to solve a problem today instead of pushing to get the most from our entire program. Even worse, it leads to the prioritization of these tests over more efficient and discipline based tests because of the immediate payoff in the “I am right” reward they offer.

The reason that this fails is because the focus is on that short term gain, we often think it will take too much effort, both resource and especially politically to try to learn in our efforts. Instead, you end up down the path of simply trying to figure out if version B is better than version A. It isn’t about figuring out what you should be doing, it is about trying to prove your idea better then the other persons idea. In some cases, groups will add a few smaller tweaked versions of A or B (which is still better), but end of the day, your test ends up answering, “Which is better, version A or version B”. If the concept is bad, it is an inefficient test. If the section you are optimization is not as important, version B can be massively better then version A, but still only as valuable as the least “better” version of a different section of the same page (or a different type of change to the same item), it is still an inefficient test. It is far more valuable to know that if I spend $5, I can get $15, $7, $20, or $30 then to just know I got $7 and be happy with it. True optimization is to figure out the best path, not to just measure the one that you are already on. Yes, you get a result, but it is a really inefficient result, and often leads to further inefficient results as you continue down that path. Even worse is when you try to force a MVT to just throw multiple better test together and simply increase the speed of your sub optimal outcome. There is no way to make this type of testing efficient, unless you get ahead of the problem and can deconstruct the question of the test.

Best testing takes a different approach, it asks much more fundamental questions, such as “Does my copy matter the most on the page?”, or “which factor of the button is most influential?”, “what are the feasible options”, or even “does changing the button, is that the best place to put my resources?”. Best testing is about figuring out where the best places are to focus your energy and the best feasible alternative; democratizing the entire process so that your outcomes are less biased by any individual or idea. Best testing becomes about the system you have in place to make decisions, and not about any individual idea that goes into it. Any system is only as good as the quality that goes into it, but it designed to maximize your returns that come out of that system. It also makes you deconstruct your questions, one of the single most important skills for a successful program, and challenge core assumptions and biases that you have. The key is to figure out what places and what types of changes are BEST relative to each other, so that you can align resources and internal thought towards those points, while eliminating waste on things that do not matter. The act of valuing possible actions against each other with causal information is the single greatest way to maximize resources, not use them unnecessarily. It challenges you to look at the page and user experience as a holistic item, and that each component is part of that process, and then to weigh the value of each one AGAINST each other. It isn’t about your idea, or even any individual idea any more, it is about creating a system by which you can learn and figure out where to focus your energy to get even better concepts and even better outcomes. Who cares if the copy improves the page if it only does it at 1/10th the scale of changing the background, or the main image, or that small section on the page that you never expected?

What is amazing is that you are left with two options from the BEST testing:

1) You were correct – In which case you have confirmation and you still get the same lift.

2) You are wrong – You learn valuable information, and you get MORE lift then what you had before. Even better, it might send you down a path that you weren’t even considering before.

What happens when you are stuck doing better testing, is that you get the short term return, both on your idea, but also politically by showing that you were correct. You are ignoring the long term opportunity cost of figuring out what matters most and building off of something. It gives you a shiny object that you get to show off to anyone that will listen, but the real question is what are you giving up going down that path?

The irony of this is that everyone always thinks that that will take too much time or resources, when for a large majority of sites out there, it is almost always the exact opposite. Being efficient means that you are testing based on your current resources and the way to maximize speed, where better testing often leads to forcing an overly technical solution in order to execute the one idea that you are willing to consider. What is true however is that you have to shift how people think, and challenge them to understand some new disciplines, in order for them to accept and execute in this way. It takes a culture and a person willing to be “wrong” and one who is willing to let go of their ego for the sake of the site. To paraphrase Kathryn Schulz, “you are already wrong, the difference is simply that you will know that you are wrong”.

What you get out of your optimization program is all about the system you put in place and how much you are willing to challenge yourself and others to find the best thing you can be doing. It is easy to want to take the immediate payout of being right or proving a HiPPO right, but at the end of the day, you will always be better off if you focus on the disciplines that make you successful. The best thing that testing can do is challenge or eliminate human biases from the plate. It can create an equal playing field so that you can correctly know the causal value of an item, and to be able to measure the EFFICIENCY of improving it. It allows all ideas to be measured against themselves, and more importantly, it stops groups from spending hours talking about ways to improve things that don’t matter, while teaching what does matter and how best to impact it. It doesn’t eliminate the value of good input; it just filters it and refines it so that it is spent on the correct things, not just what person A thinks is the most important thing.

In my quest to tackle the major biases that plague our industry, the second one I want to tackle is Expectation Bias or “the tendency for experiments to believe, certify, and publish data that agree with their expectations for the outcome of an experiment, and to disbelieve, discard, or downgrade the corresponding weightings for the data that appear to conflict with those expectations“.

In statistics and math, you deal with the concept of expected outcome. You take the payout, factor in the percent chance of it happening, and then factor in the number of chances you are taking to reach there. This is the exact reason that the lottery is such a bad investment, because it has a massive outcome, but such a low likelihood of reaching that conclusion that your expected payout is never going to happen (Neglect of probability bias). Just because one person did win the lottery does not mean that most people win the lottery. The same is true for just reading the stories or copying others. There is a want to push out and tell people about how advanced you are, as well as the want to see immediate payoffs (hyperbolic discounting). There is also the need to distinguish yourself or your offering and to discount others if an action only works 1 out of 20 times, and even then you don’t know if it is the best outcome even for that group, what possible good is it going to do you?

Obviously, the problem then becomes knowing the likelihood and the relative value of the outcome…

Good luck getting that information if you don’t have direct interaction. Not only are people hesitant to share if they know, but people are inherently wired to not seek or know that information themselves (choice supportive bias). What does happen however is a giant group of ankle biters… people or groups (especially agencies or “experts”) who promise that they have all the answers, who tell you that if you listen to them, everything will be golden, and that all of their clients are super successful. It is not evil, it is not even that they know what they are acting selfishly, they are just wired to think in these terms, as any human is to want to believe them. This phenomenon and the play of it on the human psyche is a recurring theme and one that leads to very inefficient and under performing programs, where people are left either defending poor results or seeking yet another magic answer to all their problems.

Successful consulting and successful programs focus on the disciplines of success, not just the actions. They talk about both good and bad outcomes, and how to get more of one and less of the other. They focuses on what defines successful actions and alternatives, not just what you are doing or a single possible alternative. Challenge anyone or anything that only talks about “success stories” or who can’t talk about how often things work and how that compares to alternative processes. There exists a feedback loop where bad practices are shared and championed just because they are being acted on, and not because they are valuable. It allows them to tell stories that show that they got someone to do… something. It is up to you to stop your path down that mobius strip, and instead challenge yourself and others to think differently and to think in terms of finding a better way and a better answer. Programs have to be able to learn, grow, adapt and deal with real world problems, which means that any single “answer” is just playing to your hubris and leaving you open to future failure.

One of my favorite sayings is “Information is not knowledge”. Seeking out better information and challenging the stories and practices of others, challenging your self to learn and grow and change your biases and your want to be “right” is the only way for your program to truly move forward. Focus on the system, focus on learning, focus on fixing why you do what you do, and never stop, never let go, and never let yourself fall for these traps. That is how you succeed.

One of my favorite online blogs is You are Not So Smart. I find it fascinating the fallacies that make up our day to day lives, and find that the number one driver of changing client cultures is to tackle and teach lessons that help them challenge or break some of these fallacies.

The first of these biases is CONGRUENCE BIAS, or “the tendency to test hypotheses exclusively through direct testing, in contrast to tests of possible alternative hypotheses“, or otherwise known as, trying to prove myself right. You see this all the time in testing groups, as they test only 1 or 2 things that are very similar, then get a small win and claim it is the greatest thing since sliced bread. You also see this when they know they are going to test targeting to return users, or changing the product image on their landing pages, or any predetermined priority of elements or changes. We are all guilty of this, and it takes discipline and dedication to not let it run your program or your life.

There are entire sub industries who feed and live off of this fallacy, like sites that show winning tests, many “top” blogs, or people or agencies who sell “we have all the answers”. Even internally, how often have you sat around having “I feel/I believe” conversations to determine what the best course of action is (and as a side, why do people think those two statements mean something different in that context?). You also see the same thing when you have groups who all they want to do is talk about tests, not about their program. It means that they are so caught up on trying to prove themselves right that they are missing the forest for the leaves.

It is fascinating how often the groups that get some of the worst ROI for their testing are the ones who champion and talk about it the most, and the reason for it is this bias. The people in charge are trying to make themselves look good (I had the right answer) instead of finding the best answer. The answer is simple… know you are wrong… even if you have the perfect “best” answer today, you are going to be wrong in a number of months. If you don’t have the context for the result you have, or if you don’t hear the context for the result that someone is pushing on you, then you do not know anything about the value of the outcome.

If you want to make sure that you are both getting the most value, and that you are letting a program learn and grow to the best conclusion, the choice is simple, challenge this bias. Include the null assumption, include every feasible alternative, even if you think it won’t win (there is an almost inverse correlation between what people think will win and what will win). Challenge that just because one person out there got a 10% lift doing something, that it is the best thing for you? How do you know how many other people failed miserably doing it? How do you know that even if they got the 10% that they couldn’t have gotten 15 or 20% by doing something simpler and with less effort?

Challenge yourself… Challenge how you think and challenge how others think and you will find the next “truth”. Let the reality of the data (the CASUAL data) tell you where to go and what to do, don’t let hubris or popular opinion tell you the value of something.

The system is what gets you values anyways, not any individual an action.