The many layers of what can run a testing program off track are both complicated and simple. All the major errors come from a need to fit an existing structure, to do what your boss wants, and most importantly to do what will make others happy. All the hard work that really defines success is the things that no one wants to do, be it agreeing on a metric, being the leader that your organization needs (even when they may not want it), or making sure people understand efficiency. All of these sins really are about defining how you are going to act when it comes time to do an action.

The next sin, only testing what you want, is the first one that is really about the action itself. It is the act of not just sitting together and “brain storming” or listening to a pitch about something that sounds great, and about incorporating the need to grow and learn in your actions, so that the path your group takes is organic and not inorganic. Groups get so caught up on only testing what the boss wants, or what your design people think will win, that they miss almost all the really important successes that can happen from a test. Every group starts out by wanting to test out one feature or another, and everyone hears about a best practice or a cool thing that another group did and they want to do the same. We fail to feed the system with a broad range of inputs because we can’t see past our own opinion, and because of that we dramatically lower the outcomes of our efforts.

Groups fail when they are too caught up on what wins, or on proving someone right. People are so caught up on validating an idea that they fail to see what other options will do, or even more, what if they are wrong? What matters more is not the discovery of a single validated idea, but the comparative analysis of multiple paths. This means that the worst thing we can do is limit or focus any effort only on what we want or limiting our efforts to what is popular. The sin in testing is the want to validate instead of learn, and the want to limiting of focus only to our own opinion. The truth is that the least important part of any test is what wins, since the value of the “win” is only as valuable as the context of that win. If we discover a 5% lift, that may be great, but if in the same test we could have had a 10%, 20%, or a 50% lift, then the 5% suddenly becomes an awful result. We get the most value when we are wrong, and when we discover this and allow ourselves to move down that path. Being aware of your ego, and not limiting your efforts to you or your bosses opinion, is what defines the magnitude of actual value you are achieving.

The sad truth is that we are often extremely unaware of the real value of our opinions. There is almost an inverse correlation between what people think will win, and what will win. One of the best ways to test this is to make sure that each test has a large number of very different variants and then do a poll before the test for what people think will win. You will find almost no connection between votes and outcome, which says a lot about our ability to measure things rationally after we have formed an opinion. This means that anytime that we only test what people think will win, or what they want to see will win, we have fundamentally crippled our ability to deliver meaningful value. Remember that if you only test two things, and the thing you want won, all you have done is added cost with the test. Challenge yourself and others to think in terms of possibilities and not opinions, and to scope things in terms of achieving the most options, and not just the set options.

One of the hardest tasks for groups to deal with is the need to assume that they know nothing about the value of an action. You are invested in proving to others that you know best, or in the value of any action that is already happening, but the sad truth is that most actions are done out of pattern and history and not because of measured value to an organization. Even worse, we build out giant project plans that we suddenly become inflexible to change or disrupt, being so focused on completion that we only pretend care about the actual value of the project. The need to take a step back and understand that, “if I am right, it will prove itself out. If I am wrong, it will show me a better way to do things” is easy to say but almost impossible to take hold of immediately. There is nothing worse then doing a massive project only to discover neutral or negative performance, yet this is by far the most common outcomes for groups when they test out large redesigns. It is vital that for programs as they build out that they not only test what they want to win, but that they build into all plans dynamic points where things can go directions not expected, so that we are not so inflexible to the reality of quantitative results.

Ask yourself these questions before you take any action: “how do I prove myself wrong?”, “what if I am focusing on all the wrong things?”, “what are the other feasible alternatives for this page, section, module?” It sounds counter intuitive, but it will help you understand just how much larger the testable world is from the world that you inherently would start out with. You are not limited to only what you want, you are only limited by your imagination and the efficient use of your resources. Force every action to fundamentally deal with these questions, and not the question of “how much better is this idea?”.

It is easy to limit the possible value of your program from thinking you are right. All people are wired to do so, and build their empires off the projection of this knowledge to others. Building out your tests to get past this sin, and instead find the right answer and to know how it measures to the larger world is vital to the level of value that you can get from your testing program.