Building a true optimization program in your company can be a daunting experience. No matter how much you might want to make things work perfectly, the newness of the concepts presented, the politics around who is “right”, and a hundred other factors conspire against you. Most people speak about wanting to get good results, but are often unwilling or incapable of changing their own behaviors, let alone others, in order to get those results. Even worse, there are very few people who have actually built a world class program, and they are drowned out in a sea of “experts” who have the one thing you need to do to succeed. With all that information, where you are mentally about your program speaks volumes about the value you are getting and what the next steps are to really become world class.

No program is perfect day one, and almost all of them have to go through some very difficult growing pains before they are even functional. It is true that every program follows a similar pattern of evolution, but all programs risk eventually stopping their evolution due to a lack of will or understanding. The challenge starts with the mental evolution of the program, since the functional parts are mere reflections of where people view testing. It is important that you understand where you are, and where you need to get to in order to succeed.

The challenge is that all programs reach a stopping point, either through mental exhaustion, political pushback, personal ego, or a hundred other reasons. The key to becoming a top program is to get past that point and continue down the path, even when it seems daunting or does not seem to help you advance politically.

The mental evolution of programs:

Random Testing –

All groups start here, thinking of testing as a one off action you take to figure out which piece of creative to show, or which landing page is best. No program is able to achieve the efficiency that is necessary for their program when they are stuck at this phase, yet most conversations around testing and a great many programs never get past this point. One of the key reasons for this is the comfort and the easy to grasp nature of this stage. This is what most people think of when they think testing, and that is a shame since so many will never see the power it can truly bring to their organization.

The key signs of this stage are: “better” testing, each test is an individual project, you need to get approval for each test concept, you have no rules of action. Fundamentally you are focused on finding out who or what is “right”. Testing is a one off project that you do when you need to make a decision. Put succinctly, if you are talking about what you want to test, instead of letting results tell you what to test, then you know you have not moved past this phase.

One of the other major signs of this stage is the lack of aligning on site goals. If you have not gotten alignment on a single success metric for all tests, then I can guarantee you are at this point. If you are stuck thinking about specific metrics for a test, or think that you will decide how to act and what is important when you get the results, then you are firmly stuck in this phase. If you try to think or act on the data from tests the same way you do the data from analytics, you can not ever move past this point.

It is possible to get value at this stage, and the sad reality is that most groups never leave this stage, but ultimately you will never have a real optimization program and will be getting pennies on the dollar return if this is where you keep the conversation. Groups that are stuck in this way of thinking often think that more tests means more value, and that is true if you are leaving the outcome of a test to random chance. Groups that want to be efficient and to get real value from their program however need to apply those resources not towards running hundreds of tests, but instead towards shifting the mindset of the program to insure higher returns and more long term value from each and every action.

Long Term Site Integration –

The next evolution is to start tying in testing into larger projects. Working on a redesign? You start testing out smaller portions on the way. Focused on personalization? Then you start testing out different pieces of content or you start testing for different segments. Testing has shifted from being a random action of choosing between to choices to one that can shift and change the entire direction or path of a project. To reach this stage, you must be willing to shift some part of the project away from what you want to do, and instead choose to do the things that the data tells you to do.

The benefits of this stage are the start of organizational building blocks that are fundamental to a successful program. You will have to have agreed on what is success, you are starting to look at testing as an efficiency tool, and you will have built out some processes to make testing more efficient. Most likely you have some more dedicated resources and have testing as an ongoing thing, one that is not just a novelty one off. You are stopping testing what you want, and are letting some results from the test determine the path of a project or initiative.

The limitations are that you are just doing more of the same. You have not really built out a full program and you are still focused on “better” ideas. You are just creating more structure for the randomness of the previous phase, and it is starting to shape a direction, but you have not bought in to testing as a means to the question instead of just a way to find an answer. All groups have to get through this stage, and the ones stuck here are going to get more value from the random testing of the earlier phase, but it is important that you focus on moving to the next stage, which is the largest divergence on the list.

Disciplined Base Testing –

This is the real litmus test of programs and the largest gain and divergence point. Very few groups make this leap, but the ones that do see testing as a very different and more valuable component of their entire organization. The keys to this phase are a movement away from “better” testing and a change to open ended looks for the most efficient ways to apply resources. You are no longer looking to test out what you want, but instead using testing to understand the value of different alternatives and letting the results dictate the path of your tests and your initiatives. All tests do not predispose an outcome, instead focusing resources constantly towards the most efficient answer from the prior actions. Test ideas and appeasing CXOs becomes secondary to the discovery process and the opening up of testing to dictate its own path.

A sign that you have reached this stage is that you no longer look at just a test result, but instead focus on the value of outcomes relative to each other. You measure outcomes by the value of relative actions and not just that you went from 1 point to another. If you are not proving yourself and others wrong constantly, and if you are not humbled by the fact of how little you really know, then you have not reached this point.

The benefits are a constantly growing understanding of your site and users, and a move to ensure that all efforts are focused on the most influential sections and in the most efficient manner. You are no longer worrying about a “roadmap” or about what won, but about the process of figuring it out and constantly acting. You are starting to build out the trust that the system is only as good as the input, and to not worry about who is “right” as opposed to providing quality differentiated alternatives. You are learning with every action, and you are constantly stopping current paths that your organization is on and the causal data is discovering the value of new paths that you never thought of or would not have normally pursued.

The limitations of this stage is that you are going to upset a lot of people. You are going to be constantly proving that what people have held dear and believed as the core to their benefit to an organization is actually negative. You are going to show that myths passed down for years from schools, experts, and the very thing that CXO people hold dear is wrong. If you have not built out a culture where you are focusing on being wrong, where the goal is actual success and not propagating someones agenda, you will be dealing with constant headaches and internal strife. It takes a special type of person to stand up to pressure and to do what is right, even if it is not always in their best interest. If you are not willing to pursue results over “glory” then you will most likely not ever reach this phase in your programs growth.

If you have dealt with those issues, even if with just one or two groups, you will see dramatic improvements to the efficiency and return on your efforts.

Constant Iterative Testing –

There are very few organizations that have reached this point. At this phase, all parts of the site are open and constantly evolving using discipline based testing methods in order to grow and to get more efficient. There is no longer a view of a project at all, but a constant use of multiple resources to have a shifting and evolving user experience. All jobs in the organization have optimization as part of their duties, and no longer are you having debates about what you should test and what you think is better. Everyone is aligned on a common goal of growth and of proving each other wrong, not right. It is not about the test proving anything right, but instead about the quality of the input that is used to feed the system, which is starting to dictate just about every part of your user experience and internal resources and initiatives.

Getting a program to build out the mindset that allows for this phase usually means that you have dealt with all of the negatives that might arise, and have people aware of the benefit of being “wrong“. There is nothing that will show the inefficiencies of your organization faster then trying to constantly do things that might “hurt” someone. If you are not willing, capable to deal with the previous phases, most likely you will not come close to this level in your program.

Optimization Organization –

I refer to this as the mythical unicorn, as there are no organizations in the world that have reached this “nirvana”. At this point, all concepts feed the system and the system dictates the outcome. This is about letting go of worrying about who is right, and instead understanding that you still have to feed the system, but making the final decision has to be left to a disciplined (non-biased and predetermined) use of data, especially causal data, to make those decisions.

No group has reached this point because all organizations are run by people, all of which have their own agendas and all of which need to prove themselves “superior”. I don’t expect there to ever be a point where people are capable of putting their egos and politics at check, but working towards this point is the only way to really make a true difference and not just use data to push an agenda or to promote yourself.

The evolution of programs is a tricky thing, and not one that is quite as black and white as this path might make it seem. What is important is that you understand that you have to shift how you think, and be willing to change all the pieces that follow, in order to be successful. If you are still thinking about testing as a means of choosing between two items, or if you haven’t built out a culture where being wrong is more important then being right, then all the resources in the world wont make you successful. Success is not a random thing, and it is not dictated by how many resources you have, but instead by your willingness to prove yourself and others wrong. Building out the right mindset determines the long term value you receive, yet very few take the time to really understand or educate others. There are plenty of material out there that is happy to make you feel good about whatever stage you find yourself at, but if you really want to get value and really make a successful program, nothing will top constant hard work and the willingness to challenge the norm and to do things against “best practices”.