One of the most common refrains from established testing organizations is the need for process. They use the “need” of improved process as the excuse for why they don’t do more, why it takes so long to run tests, why they only get ok results from the program, and why they don’t impact the business in a bigger way. They love to talk about how many tests they can run, but are hesitant to talk about the real impact of the program. Process becomes this constant refrain which serves no purpose but to explain that persons existence and to insure that the no one gets upset. It doesn’t matter what is going on as long as consistent action is happening and other people are using it, as if just the act itself is the provider of value. Process gets turned into this mythical jabberwocky whose function is simply to make it impossible for programs to rule the world. Why do groups so easily accept process as the cure all for what ails them, instead of focusing on improving the efficiency of their program?

Process is just a pattern of actions. It’s what those actions are, and how they are leveraged that truly matter. A great process makes it easy to do the right things quickly and consistently, but it also enables you do the wrong things consistently as well. So many groups are limited because they get to a point where they are proficient at running a test, but have not spent the time to learn and accept what makes a real methodology or strategy to testing. They thinking in terms of how to make their lives easier or how do I do more, not about getting more from what they are already doing. They accept the first answer they find and are more then happy to propagate a myth simply because it gets people to do more. People are so happy to get a result from their test that is actionable, that they try to recreate ways to do that more and more, instead of focusing on ways to get better results or to increase the likelihood of a positive outcome. Groups in that stage are forced to try and run more and more tests in order to increase the value of the program, confusing excess resource usage with value.

It might seem frustrating because you want to do more, but the question is never how do you get more resources or how do you run more tests, but how do you get more from what you do have. Running a test when you start seems like a Sisyphusian effort, but the question should not be how do you make the boulder smaller, but what is waiting for you when you make it to the top of the hill. It is not random, nor is it impossible to get great results consistently, but you have to change how you think about the world before you can change the hill you are trying to climb. You have to stop trying to impress or make people happy and instead truly focus on the discipline of making each action more valuable then the last. To top it off, there are a large number of groups who convince themselves that the larger hill must have a better reward, when often the inverse is correct. Climb the right hill, and the path is shorter and the boulder smaller. Climb the wrong hill, and all you are looking for is a smaller boulder to get off your shoulders.

This is not to say that a proper process cannot dramatically improve a program, however process becomes a problem when held up as a holy grail as opposed to just a means to an end. I want to walk you through a mental exercise where we evaluate four possible scenarios for an organization, and see what the outcome would be. In this case, we are simply dividing groups who have proper process in place and those that have proper discipline and methodology in place.

Group #1: Poor Process & Poor Strategy

Groups in this quadrant suffer from resource shock for testing. They have accepted that they might only see results anything positive from 25–50% of their tests and are struggling to run a number of tests. They are often frustrated by their inability to do more, and as such they try to make up for it by making larger and more complex tests, often requiring even more resources at each turn and slowing down the process even more. They are often running tests in-line with other groups in the company and trying to piggyback their efforts to show that they add value.

Most organization start here and sadly never leave, even when they dump resources into hiring more people or hiring an agency. Groups in this phase suffer from inconsistent returns, low value, and poor adoption in the organization. The only saving grace is that they are saved by the low amount of resources that are dumped into the program and by the small exposure highlighting their inefficiency. To make it worse, agencies love to keep people in this state, as it makes their efforts to get a result seem much greater then they really are and it justifies massive amounts of hours to achieve these minuscule results. They get one or two results over long periods of time and then spend hours justifying it as a great example of the power of their efforts.

At this point, you are a weak man trying to push a rock up a large hill that you are not sure what is on the other side of. It can be done, but it takes a long time to succeed.

Group #2: Poor Process & Proper Strategy

Organizations in this phase have very poor resources, low communication with other groups, and only the most basic of infrastructures on the site. They have to fight IT and product management and are often stuck running a few tests a month or quarter. The tradeoff is that they are maximizing their return through proper use of resources, learning, and best not better testing. They are at a state where they have lots of roadblocks and low resources available to them, but they are planning their program around the most efficient ways to exploit the resources they. They are not settling for the excuses, nor are they running a perfect program, but they are focusing the testing program now that they have instead of trying to make what little they have fit into a predetermined mindset of what “success” is. They are focusing on using what they have to its maximum return instead of facilitating others ideas or “test ideas”. They are not trying to fit a square peg into a round hole, but instead trying to find what fits best into the hole they have access to.

At this stage, you expect that every test they run will give them multiple pieces of information about the value of feasible alternatives against each other and expect that 100% of tests will feed the next test and produce actionable insight. They expect that 75%-95% of tests will also deliver actionable meaningful lift of multiple alternatives which they can then use to measure the efficiency against each other to choose the best option. They might only run a few tests a month, but they are usually getting magnitudes greater direct return from their entire program then programs spending 20x on resources.

Groups in this phase use very little resources and due to that are not able to move at the speed they want. These groups though are able to maximize their efficiency by focusing on getting the largest return for what they have and making sure that every action returns positive and meaningful information. They could do more with better processes and greater resources, but they are still producing results that knock other initiatives out of the water.

At this point, you are still a weak man, but you have chosen shorter hills and you have a really good idea what is waiting for you on the other side. You are not letting some person miles away dictate the hill your go over, but instead finding the best use of resources. Each hill you peak points you to the next hill to tackle instead of predisposing the order of hills that you will tackle.

Group #3: Good Process & Poor Strategy

Groups in this phase have accepted that testing is a great thing for their organization. They love to test and are great at getting tests live. They can often expect to get hundreds of tests live a year and are great at assigning resources, creating a charter, having QA go through a test, and have multiple resources assigned to get a test live and going. Often times they might also work with agencies and pay outside people to make sure that they have those resources. They are still only getting results from 25–50% of their tests, and are often finding that they have to spend more and more resources to get larger and bigger creative as they run through their “roadmap”, but at least they do have results because they have run so many tests. Often times groups in this stage have come up with non meaningful measures of success, such as click through rate or bounce rate as a measure in order to prove to executives that they are doing far more then they really are. This is the point that a large number of very mature programs find themselves.

Groups in this phase have large dollar figures they can point to for outcomes, but when looked at for efficiency are often getting very poor ROI compared to efficient testing programs and are taking resources away from other efforts in the organization. They have built an empire for the people running the program and because of the nature of testing, are often one of the top revenue drivers for the company, but are dumping money to do so. I love that these groups appreciate and believe in testing to the degree that they do, but in many ways not maximizing their investments. By propagating the myth that more tests equals better results, they allow people to confuse getting a result from a test with getting a meaningful result of viable alternatives. Often times these groups get about the same monetary return from their programs as the organizations in group #2 above, but are spending magnitudes more on resources to do so.

At this point, you are spending money for a giant crew and large industrial equipment to push the same rock up usually now steeper hills where you still don’t know what is waiting for you on the other side. There are lots of groups out there waiting and asking to take your money to be the crew and to give you the tools so that you too can mount this hill.

Group #4: Good Process & Proper Strategy

There are very few groups that are in this quadrant, but those that constantly see returns and have built out their programs so that they can test everything and have everyone on the same page to act and test what is necessary. Because they are being efficient and learning as they go, each test exponentially has a higher chance of a positive return and they are hitting on 90–100% of their tests to generate at least 2–5% lift and to compare alternatives. They will run a number of tests, but they focus on the outcomes of the tests and not the raw number it takes to get somewhere. They are also changing the path of the entire organization, showing the causal value of different alternatives and helping to stop current activities while discover new ones.

Groups in this phase see magnitudes higher gross return on their efforts. They are often not spending much more in the way of resources then organizations in group #1, and they are making high multiples of the gross returns of groups in group #3.

At this point, you have gotten some people that know how to move a boulder, and you have bought some equipment to move it, but you are chosen much better hills with much more consistent returns and you are choosing to point the people only at the hills that have the highest returns. The boulder has become irrelevant to the hills you are tackling.

A proper process and more resources simply highlight the dimensions of your program. If you are inefficient or do not know how to run a proper program, it allows you to do more and get a higher gross return. There are many people who hide behind process in order to justify their jobs, and that is a shame, as they could do so much more if they are just willing to play less politics and focus more on achieving results. If you care about being efficient and about getting the maximum amount of return however, then you have to focus on both parts, and understand that more is not always better. If you are only hitting on 50% or less of your tests, or you only get a single outcome from your tests, then you have to run more than double the number of tests in order to get the same gross return of someone who is hitting on 100% of them (compound returns and efficiency of measuring multiple alternatives against each other).

It might seem frustrating because you want to do more, but the question is never how do you get more resources or how do you run more tests, but how do you get more from what you do have. It is easy to blame process or to think that a better process somehow creates a magical panacea that solves everything, but that is because we are looking externally. It is far easier to blame others then to look internally and make sure you are doing the most with what you have. Be honest with yourself and use what you have, and you will be amazed at the results you achieve and how little in the way of resources you need to achieve them. Change how you and others think first, and then process can truly make a big impact.