One of the great truths about any organization is that no matter what it is you are doing, each program eventually plateaus and finds a normalization point where it no longer grows at the same rate or with the same push it did before. Whether it is mental fatigue, new objectives, changes in leadership, or more commonly reaching the end point of the current path of thinking, each program can only go so far forward without a re-invigoration of new ways of thinking and by challenging itself to get better. It is only by bringing in new ways of thinking and challenging core beliefs that you are free to grow past those self-imposed limits.

In our introduction, we talked about disciplines that enable you to move faster and align on a common goal. In the second series, we went over disciplines to help you think about tests and testing differently to get more value from your actions. The third and final evolution takes us in new ways to view the world and our organization, and challenges us to go in new directions, to understand new paradigms that should fundamentally challenge some of the most common and fundamental beliefs about data and optimization.

One of the great quotes that I keep close at heart comes from John Maxwell, “If we are growing, we are always going to be out of our comfort zone.” With that in mind, I want to introduce these paradigms for your program and challenge you to take these and evaluate them outside of the fishbowl, as an idea on its own that can help you program get past its current plateau and to help you grow in your thinking about optimization.

Analytics and Optimization as very Different Disciplines –

There are many different ways that you look at and act on data in testing that are the exact opposites of analytics. Where so many programs fail is when they force one way of thinking onto their data, resorting back to what they are most familiar with. In the world of analytics, you have to look for patterns and anomalies, look across large data sets and try to find something that doesn’t belong, and that doesn’t fit with the rest of the data. You are constantly looking for outliers that show a difference and then extrapolating value from those measurable differences. In the world of optimization, you have to limit yourself from looking at anything but what you are trying to achieve, and to act on data that answers fundamental questions. It becomes extremely easy to fall back into more comfortable ways of thinking, because the data sounds and looks similar, but ultimately success is dictated by your ability to only look at the data through a different lens. You have to stop yourself from trying to dive down every possible data set and instead focus on the action from the casual relationship around the single end goal.

It is what you don’t do that defines you as much as what you do. It is about “did removing this module improve revenue performance“, not “did the CTR drop of this change increase CTR to the main image and where did those paths lead“. It is also about not allowing linear thinking to interrupt what you are doing. You have to focus on the value of actions, not the rate of them. You are looking at the value add from a user (RPV), not the amount of short term actions (CTR). Never look at how many people moved from point A to point B, but instead only look at the measurable impact towards your site goals. Just because you increased clicks, or got more people into a funnel, or even got more transactions, it does not mean that you increased revenue. Assuming there is a linear relationship between action and value can be extremely dangerous and myopic, many programs have been ran into the ground because they do not understand the difference between the count of and action and the value of the action. Analytics forces you to think in terms of rates of action, but optimization forces you to think about the value of actions and the cost to change a person’s propensity of action.

Think of your site as a giant system. You have an input of people, with each input type interacting differently with the system. The things you sell, the layout, the experience, all of it makes up a giant equation. When those people enter your site, they go through, and they come out the other end at some rate or some value. The numbers or rates associated with that one path is analytics. That inherent behavior based on the current user experience is their propensity of action. In testing, you have to solely focus on your ability to increase or decrease that propensity of action, not about the absolute value of that action. We care that we increased that behavior by some delta, some lift percentage, not that it was 45% and moved to 49%, but that we increased it by 8.9%.

In testing, the answers you might receive will be of the nature of “we got more of the high value and less of the low value populations” or “the system improved as a whole by 4%.” Ultimately “answers” matter far less then the changes observed and your ability to act quickly and decisively on it. What you won’t receive is why, or what each individual part of the system did to get you there. Those answers are stuck to the realm of correlation and as such have to be ignored because you at best have only a single data point. We are trying to move forward as quickly as possible in the realm of optimization, so getting lost in loops of trying to answer questions that are not answerable only hinders your efforts. No matter how much you analyze an individual test result, you will never have more than a correlation. This means you have to think differently in order to use that data. It doesn’t matter why, or which piece, or even which individual population (though dynamic experiences on outcomes is important) so you have to force yourself to not go down those roads.

It is also about the ability to hold yourself accountable for change. So many analysts fail because they view their job responsibility ends on the moment they make a recommendation. There is a revolution taking place in our industry, lead by people like Brent Dykes, that is changing the entire view of optimization away from the recommendation and data, but to the final output. In optimization, you are only successful if the results you find are acted on and made live. It requires you to view the cycle as one of action and not one of inaction. It is not that both don’t have their place, but you to be really successful you have to be able to step away from your analytics self and instead think differently and force yourself to act differently in order to get the results you need.

Testing Applied to Multiple Teams

Testing is something that has many core disciplines, but takes on a very different look and value for different groups. Your IT team may get a completely different “value” from testing than your merchandising team, as your design teams might from your analytics team. Many groups believe that because they have applied their testing from their landing pages to their product pages, that they have expanded the value of testing throughout the organization. Instead they need to rethink how optimization disciplines can interact with the different groups efforts on a fundamental basis. Testing is not just changing a marketing message, it is the evaluation of possible feasible altneratives, something that all groups need to do to improve their effectiveness. Testing is just as applicable to your SEM team, your merchandising, your product management, your IT team, your personalization team and many others. Each group has different needs and different disciplines, and as such you have to apply the disciplines of testing to them in different ways. A IT team can use testing to decide on which project to apply long term resources to. Your UX team can tie testing to their qualitative research to understand the interconnection of positive feedback to overall site performance. Your SEM team can use testing to measure the downstream impact of their various branding campaigns.

The reality is that applying all the unique benefits of testing to different groups, and not just increasing the space that you do the same things to can fundamentally improve your entire organization. While this might sound like a simple one, the reality is that most groups do the same type of testing or try to apply the same techniques across multiple parts of the site, not for different teams. Each group may be aligning on the same goal, but they do things in a very different way. Applying optimization to those groups looks and acts in very different ways, and such it is difficult for most groups to really apply these disciplines in a way that truly impacts the fundamental practices of more than one group.

Instilling this use of testing as a fundamental building block also allows you to get ahead of a large number of major problems. It forces organizations to test out concepts well before they decide on them as long term initiatives. One of the most common examples of this is in the realm of personalization, where so many groups are sold on the concept, but not willing to go through all the hard work of figuring out exploitable segments or the value and efficiency of various ways of interacting with the same user. Getting ahead of the curve and testing out the efficiency of the effort will save dramatically improve the performance of the effort. If you test out a complex idea in one spot against other feasible simpler ideas, and find the simpler idea is better performing, as it almost always is, you save massive IT resources while getting better results. It is far more likely that simple dynamic layout changes for firefox users are going to be magnitudes more valuable then a complex data feed system from your CRM solutions, and testing is the bridge to know that before you fall down that rabbit hole.

Each group tends to end up at the Nth degree of the same thing they bought the tool for. So often, the fear of the unknown or of challenging someone’s domain stops new groups from allowing testing in, but when you can overcome those barriers, you can have an exponential impact on the organization. When you start trying to apply optimization to multiple types of internal practices, and you are able to bring the results together in a real synergy, that is when you are able to really see optimization spread and to see the barriers drop throughout an entire organization. It also the point where those lessons you learn become three-dimensional and become universal across the entire organization.

Testing has No Start and No End –

Optimization is not a project. It is not something that is just one person’s job and it is most definitely not something you can just choose to end some random Tuesday. So why then do people view it as a series of projects, with a start and a stop? Why do they view it is only part of one person’s role or responsibility, or something that is done when they have the chance. There are functional reasons to have set people assigned to testing, and as programs grow to have a separate specialized team, but that is not the end of the battle. Why do we try to force artificial time constraint on it, with starts and stops and talk about it as something we did, or will do. It is either an action that you live, or it is not. If everything your organization is doing, be it some small tweak, or a redesign, or the release of a new feature is not viewed as part of an ongoing process, with lessons to learn and to be evaluated democratically through the system of optimization, then optimization has been allowed to have this artificial start or stop just to appease various members of your organization.

Optimization has to be something you live. You have to be thinking in terms of it every day, you have to view each task as something that can get better, you have to view each idea as just one of many, and that it is not up to the HiPPo or anyone else to decide on. It is a responsibility to not let projects, or holidays, or new CMOs or anything else stop you from this constant quest to improve, the site, the processes, the people. Do not confuse the actions of running or a test as the entirety of optimization. It is vital that you view the act of creating something new as just the very first step, and not the end point. There should be no point where anything is thought of as “perfect”, or “done” or that you can just throw something live and walk away. Optimization is part of every process, it is part of every job, and it is something that everyone works together to make sure that it is part of every action that the organization takes.

When you have finally started to incorporate testing into your organization, all projects will view it as another natural part of their evolution. Project plans will incorporate not only the concept of optimization as an ongoing basis so that it is part of your expected timeline, but they will also stop trying to get everything “perfect”. If you view your projects as never finished, then there is no need to have everything get perfect signoff, nor do you need to have perfect agreement on each and every piece. What is important is that you spend as much time and resources on testing out all those ideas that you have discussed, instead of just sitting around a room and compromising on a final version. You will no longer be so caught up on your pet project, as the entire concept is that it will and must change.

So much of what happens in organizations is about the politics of owning and taking credit for different initiatives. There are people’s reputations and egos on the line when they propose and lead dramatic changes, especially redesigns, for the site. If you can truly incorporate testing and optimization as a vital part of all processes, one that is not just a “project” but is part of the very existence of the site and the group, then you free people up to no longer being so tied to their “baby”. Treat all ideas as malleable and transient, to the point that everyone is really working together to constantly move the idea forward. It will can be a dramatic shift to organizations once they reach this point, but ultimately it is when groups really start to see dramatic improvements on a continuous basis.

So often we talk about not following through with each of the concepts I have brought forth, but the reality is each action is tantalizingly easy, but the real discipline, the ability to keep pushing 6 months from now is what really differentiates programs and people. Being willing to move past the barriers, put the pieces in place that make a difference, and being willing to change how you and others think are the real keys of a successful program. If you are always trying to do what is easy, or just listen to the pushers of magic beans and myths, then you can never really grow your program to the levels that are possible. Do the hard work, get out of your comfort zone, and you can continue to get better and can continue to see more and more value from your testing program.

To navigate the entire testing series:
Testing 101 / Testing 202 / Testing 303 – Part 1 / Testing 303 – Part 2

The beauty of any program or any skill is that no matter where you are, or what you have accomplished, you can always get better. There is no such thing as the perfect testing organization, or a perfect way to use data, which means there are always opportunities to grow and to get better. With that in mind, I want to present various skills and disciplines that will allow your program to grow no matter where you are at. If you have just started out, been testing for years, or are a massive optimization organization, there are always new ways to think and new skills to allow you to get more value out of each and every action that you take.

All programs are built off of key fundamental disciplines that allow all other disciplines to grow and to be acted on accordingly. All groups start out by needing to grow past the initial challenges and limitations that face your program. Most groups find a point where testing has shown a few wins, you find places to test where it makes sense, but you haven’t been able to get the level of consistency or buying that you are looking for. You are trapped in a world of explaining why you want to run a test, or working with different groups who aren’t interested in what the rest of the organization is doing. Early development programs have many factors that unite them; you might have been testing for 3 days or 3 years, and yet if you haven’t really taken the time to grow and incorporate testing beyond a superficial way, you will never achieve the true value that testing can provide for your organization.

The first thing that differentiates programs is the organizational pieces that have been put in place to help them move at a speed that will truly impact the bottom line. With that in mind, here are 5 key disciplines that will allow your program to the next level. For any testing program to work long term and to grow, you must first address the organizational roadblocks that will without fail trip up your program. The first and sometimes most vital aspect of growing your program can be adoption and the realization that this is not a once in a while type of action that you take.

Single Success Metric –

Without fail, the single greatest determining factor in the overall success of your testing program is have you unified on what you are trying to achieve and are you optimizing for the end behavior you want. So many programs fail when they try to optimize for the next page or a small action, or even worse to try and optimize for multiple metrics to make everyone happy. So many programs are less the sum of their parts, and because of this they are never able to impact the fundamental bottom line of the business at the level that they are capable of.

It isn’t just about optimizing for the goal of the site and not focusing on a individual concept, you must be able to answer all questions of this format: If A goes up, and B goes down, what do we do? What about A and C? Ultimately, you will quickly discover there is one thing that you really have to go up to be successful. It isn’t about clicks, our bounce rate, or just moving people to another part of your site, it is about focusing on what all of those behaviors are trying to accomplish. In just about all cases, the metric you will end up using is directly tied to how you make money, something like RPV, or lead conversion rate, or an engagement metric. What is important is that is global across your entire site, not just the section you think you will impact. That is what you need to focus on. You need to have everyone in agreement on how to act on the test before it launches, so that you don’t lose momentum or get lost in the internal quicksand of politics.

Nothing allows your program to be more valuable and for you to bring people together then to have everyone pointed towards the same end goal. Nothing ends internal debates faster than not worrying about clicks or sign-ups or search versus organic then optimizing for revenue and giving all actions an equal footing.

When I come in to assist groups and make sure they are getting the most value, nothing is more vital and I will refuse to go forward unless we can get agreement on what defines success and how we are going to measure it. You are often defined by what you don’t do as much as what you choose to do, and this is the seminal moment when you can take a giant leap in your program. This is the moment when a new discipline, unique to optimization, comes into place. It also allows us to treat any idea democratically and let the numbers define success.

Consistent Resources –

One of the many failings of early programs it he lack of ownership of the process and tool. Does IT, marketing, product, merchandising own testing? Whose resources get assigned and who will run this specific test? Do you have a FTE, or just a few random hours assigned to someone who has 10 other tasks on their plate? It is vital that as you build your program, that you have consistent time and resources assigned, outside of any specific test or project. The skills of testing, not just the tool, but understanding how your site works, how to move things, testing disciplines and how to navigate internal politics, all of these take experience and time. Programs that have new people jumping on board, people who run a test once or twice a quarter, or who have not clearly identified who will set-up, run, and do reporting on tests have the hardest time creating the momentum and cadence that allow your program to really blossom and to help insure that you are getting the magnitudes of value that testing can provide.

A good first step for any program is to assign a set number of hours per week. Assume that you will always be testing, and such, this won’t be tied to any individual test. As a good next step, you can make sure that any test you run can be done with that amount of resources in one week. Assigning the same people, a set amount of time, and making sure that you use it to always be moving forward will help your program achieve the cadence and scope of impact that you are looking for.

Infrastructure –

Having a proper infrastructure on your site is vital to be able to have a real program. The pieces of your infrastructure has two main components, the site code and the organizational components. A good robust infrastructure on your site should allow you to get a new test live within a day on any important part of your site without IT involvement. If you are creating testing code for each test, or if you require a separate ticket and full QA process for each test, you will never be able to move at the speed that a successful program runs at.

The other component for infrastructure is the organizational components. Do the important groups in your organization understand how you will run a test? Why you run a test? What will define success and how best to think about testing to provide meaningful answers? Are they bought in to testing or do they see it as something that threatens their imminent domain? Whereas the technical infrastructure is something that may be done mostly at the start and then revisited from time to time, the educating and working with various groups in your company is an ongoing thing and is vital for long term growth.

Rules of Action –

Once you have figured out your single success metric, do you have a consistent set of rules to define how you call a winner? How about when you have a winner, how does it go live, who is responsible, what is the timeframe? How do you build off of a test result? It is vital that you answer these questions and get people in agreement BEFORE you launch a test. Success and failure of any test is not in the lift it provides, but your ability to act on that data in a clear and expedited manner.

Groups will often write these down in a document and get signoff from different leads of the organization, or create a wiki and provide access to everyone. Each group has their own unique version of this, but it is vital that before you get to the action phase, that everyone is clear on how and why you do what you do. Nothing kills a program faster than people desperately trying to stop a change because they were “wrong” or people not willing to act on data because they were not ready to go. Never start a test if you are not able to act on the results, and rules of action are your way to ensure that you will be able to do this.

Program Champion –

Outside of having a single success metric, there may be no greater determination of long term success than having someone with some power in your organization who sponsors and pushes testing. Having a person who can navigate the political waters to enable the disciplines listed above, and to help communicate and hold people accountable for results. This person may not be involved in day to day actions, but they will often hold regular review meetings, talk to different groups, and present results to senior leadership. It is not about “owning” testing, optimization is part of everyone’s job, but this person helps get testing the attention it needs and holds people accountable.

You can look at any of the best testing organizations, and without fail you can name this person instantly and clearly. That person has bought in and while the program is often not perfect, they have helped lead it in a way that there is no doubt who sponsors it, why they push it, and how much value it brings to your company.

These disciplines represent just the first steps to really becomes a optimization leader, but they are vital in order for you to be able to grow and to facilitate the adoption of more advances skills. In the second set of disciplines, we will explore ways to think about testing to insure that you are getting more value from each of the actions which you just helped facilitate.

To navigate the entire testing series:
Testing 101 / Testing 202 / Testing 303 – Part 1 / Testing 303 – Part 2