One of the great truths about any organization is that no matter what it is you are doing, each program eventually plateaus and finds a normalization point where it no longer grows at the same rate or with the same push it did before. Whether it is mental fatigue, new objectives, changes in leadership, or more commonly reaching the end point of the current path of thinking, each program can only go so far forward without a re-invigoration of new ways of thinking and by challenging itself to get better. It is only by bringing in new ways of thinking and challenging core beliefs that you are free to grow past those self-imposed limits.

In our introduction, we talked about disciplines that enable you to move faster and align on a common goal. In the second series, we went over disciplines to help you think about tests and testing differently to get more value from your actions. The third and final evolution takes us in new ways to view the world and our organization, and challenges us to go in new directions, to understand new paradigms that should fundamentally challenge some of the most common and fundamental beliefs about data and optimization.

One of the great quotes that I keep close at heart comes from John Maxwell, “If we are growing, we are always going to be out of our comfort zone.” With that in mind, I want to introduce these paradigms for your program and challenge you to take these and evaluate them outside of the fishbowl, as an idea on its own that can help you program get past its current plateau and to help you grow in your thinking about optimization.

Analytics and Optimization as very Different Disciplines –

There are many different ways that you look at and act on data in testing that are the exact opposites of analytics. Where so many programs fail is when they force one way of thinking onto their data, resorting back to what they are most familiar with. In the world of analytics, you have to look for patterns and anomalies, look across large data sets and try to find something that doesn’t belong, and that doesn’t fit with the rest of the data. You are constantly looking for outliers that show a difference and then extrapolating value from those measurable differences. In the world of optimization, you have to limit yourself from looking at anything but what you are trying to achieve, and to act on data that answers fundamental questions. It becomes extremely easy to fall back into more comfortable ways of thinking, because the data sounds and looks similar, but ultimately success is dictated by your ability to only look at the data through a different lens. You have to stop yourself from trying to dive down every possible data set and instead focus on the action from the casual relationship around the single end goal.

It is what you don’t do that defines you as much as what you do. It is about “did removing this module improve revenue performance“, not “did the CTR drop of this change increase CTR to the main image and where did those paths lead“. It is also about not allowing linear thinking to interrupt what you are doing. You have to focus on the value of actions, not the rate of them. You are looking at the value add from a user (RPV), not the amount of short term actions (CTR). Never look at how many people moved from point A to point B, but instead only look at the measurable impact towards your site goals. Just because you increased clicks, or got more people into a funnel, or even got more transactions, it does not mean that you increased revenue. Assuming there is a linear relationship between action and value can be extremely dangerous and myopic, many programs have been ran into the ground because they do not understand the difference between the count of and action and the value of the action. Analytics forces you to think in terms of rates of action, but optimization forces you to think about the value of actions and the cost to change a person’s propensity of action.

Think of your site as a giant system. You have an input of people, with each input type interacting differently with the system. The things you sell, the layout, the experience, all of it makes up a giant equation. When those people enter your site, they go through, and they come out the other end at some rate or some value. The numbers or rates associated with that one path is analytics. That inherent behavior based on the current user experience is their propensity of action. In testing, you have to solely focus on your ability to increase or decrease that propensity of action, not about the absolute value of that action. We care that we increased that behavior by some delta, some lift percentage, not that it was 45% and moved to 49%, but that we increased it by 8.9%.

In testing, the answers you might receive will be of the nature of “we got more of the high value and less of the low value populations” or “the system improved as a whole by 4%.” Ultimately “answers” matter far less then the changes observed and your ability to act quickly and decisively on it. What you won’t receive is why, or what each individual part of the system did to get you there. Those answers are stuck to the realm of correlation and as such have to be ignored because you at best have only a single data point. We are trying to move forward as quickly as possible in the realm of optimization, so getting lost in loops of trying to answer questions that are not answerable only hinders your efforts. No matter how much you analyze an individual test result, you will never have more than a correlation. This means you have to think differently in order to use that data. It doesn’t matter why, or which piece, or even which individual population (though dynamic experiences on outcomes is important) so you have to force yourself to not go down those roads.

It is also about the ability to hold yourself accountable for change. So many analysts fail because they view their job responsibility ends on the moment they make a recommendation. There is a revolution taking place in our industry, lead by people like Brent Dykes, that is changing the entire view of optimization away from the recommendation and data, but to the final output. In optimization, you are only successful if the results you find are acted on and made live. It requires you to view the cycle as one of action and not one of inaction. It is not that both don’t have their place, but you to be really successful you have to be able to step away from your analytics self and instead think differently and force yourself to act differently in order to get the results you need.

Testing Applied to Multiple Teams

Testing is something that has many core disciplines, but takes on a very different look and value for different groups. Your IT team may get a completely different “value” from testing than your merchandising team, as your design teams might from your analytics team. Many groups believe that because they have applied their testing from their landing pages to their product pages, that they have expanded the value of testing throughout the organization. Instead they need to rethink how optimization disciplines can interact with the different groups efforts on a fundamental basis. Testing is not just changing a marketing message, it is the evaluation of possible feasible altneratives, something that all groups need to do to improve their effectiveness. Testing is just as applicable to your SEM team, your merchandising, your product management, your IT team, your personalization team and many others. Each group has different needs and different disciplines, and as such you have to apply the disciplines of testing to them in different ways. A IT team can use testing to decide on which project to apply long term resources to. Your UX team can tie testing to their qualitative research to understand the interconnection of positive feedback to overall site performance. Your SEM team can use testing to measure the downstream impact of their various branding campaigns.

The reality is that applying all the unique benefits of testing to different groups, and not just increasing the space that you do the same things to can fundamentally improve your entire organization. While this might sound like a simple one, the reality is that most groups do the same type of testing or try to apply the same techniques across multiple parts of the site, not for different teams. Each group may be aligning on the same goal, but they do things in a very different way. Applying optimization to those groups looks and acts in very different ways, and such it is difficult for most groups to really apply these disciplines in a way that truly impacts the fundamental practices of more than one group.

Instilling this use of testing as a fundamental building block also allows you to get ahead of a large number of major problems. It forces organizations to test out concepts well before they decide on them as long term initiatives. One of the most common examples of this is in the realm of personalization, where so many groups are sold on the concept, but not willing to go through all the hard work of figuring out exploitable segments or the value and efficiency of various ways of interacting with the same user. Getting ahead of the curve and testing out the efficiency of the effort will save dramatically improve the performance of the effort. If you test out a complex idea in one spot against other feasible simpler ideas, and find the simpler idea is better performing, as it almost always is, you save massive IT resources while getting better results. It is far more likely that simple dynamic layout changes for firefox users are going to be magnitudes more valuable then a complex data feed system from your CRM solutions, and testing is the bridge to know that before you fall down that rabbit hole.

Each group tends to end up at the Nth degree of the same thing they bought the tool for. So often, the fear of the unknown or of challenging someone’s domain stops new groups from allowing testing in, but when you can overcome those barriers, you can have an exponential impact on the organization. When you start trying to apply optimization to multiple types of internal practices, and you are able to bring the results together in a real synergy, that is when you are able to really see optimization spread and to see the barriers drop throughout an entire organization. It also the point where those lessons you learn become three-dimensional and become universal across the entire organization.

Testing has No Start and No End –

Optimization is not a project. It is not something that is just one person’s job and it is most definitely not something you can just choose to end some random Tuesday. So why then do people view it as a series of projects, with a start and a stop? Why do they view it is only part of one person’s role or responsibility, or something that is done when they have the chance. There are functional reasons to have set people assigned to testing, and as programs grow to have a separate specialized team, but that is not the end of the battle. Why do we try to force artificial time constraint on it, with starts and stops and talk about it as something we did, or will do. It is either an action that you live, or it is not. If everything your organization is doing, be it some small tweak, or a redesign, or the release of a new feature is not viewed as part of an ongoing process, with lessons to learn and to be evaluated democratically through the system of optimization, then optimization has been allowed to have this artificial start or stop just to appease various members of your organization.

Optimization has to be something you live. You have to be thinking in terms of it every day, you have to view each task as something that can get better, you have to view each idea as just one of many, and that it is not up to the HiPPo or anyone else to decide on. It is a responsibility to not let projects, or holidays, or new CMOs or anything else stop you from this constant quest to improve, the site, the processes, the people. Do not confuse the actions of running or a test as the entirety of optimization. It is vital that you view the act of creating something new as just the very first step, and not the end point. There should be no point where anything is thought of as “perfect”, or “done” or that you can just throw something live and walk away. Optimization is part of every process, it is part of every job, and it is something that everyone works together to make sure that it is part of every action that the organization takes.

When you have finally started to incorporate testing into your organization, all projects will view it as another natural part of their evolution. Project plans will incorporate not only the concept of optimization as an ongoing basis so that it is part of your expected timeline, but they will also stop trying to get everything “perfect”. If you view your projects as never finished, then there is no need to have everything get perfect signoff, nor do you need to have perfect agreement on each and every piece. What is important is that you spend as much time and resources on testing out all those ideas that you have discussed, instead of just sitting around a room and compromising on a final version. You will no longer be so caught up on your pet project, as the entire concept is that it will and must change.

So much of what happens in organizations is about the politics of owning and taking credit for different initiatives. There are people’s reputations and egos on the line when they propose and lead dramatic changes, especially redesigns, for the site. If you can truly incorporate testing and optimization as a vital part of all processes, one that is not just a “project” but is part of the very existence of the site and the group, then you free people up to no longer being so tied to their “baby”. Treat all ideas as malleable and transient, to the point that everyone is really working together to constantly move the idea forward. It will can be a dramatic shift to organizations once they reach this point, but ultimately it is when groups really start to see dramatic improvements on a continuous basis.

So often we talk about not following through with each of the concepts I have brought forth, but the reality is each action is tantalizingly easy, but the real discipline, the ability to keep pushing 6 months from now is what really differentiates programs and people. Being willing to move past the barriers, put the pieces in place that make a difference, and being willing to change how you and others think are the real keys of a successful program. If you are always trying to do what is easy, or just listen to the pushers of magic beans and myths, then you can never really grow your program to the levels that are possible. Do the hard work, get out of your comfort zone, and you can continue to get better and can continue to see more and more value from your testing program.

To navigate the entire testing series:
Testing 101 / Testing 202 / Testing 303 – Part 1 / Testing 303 – Part 2