The beauty of any program or any skill is that no matter where you are, or what you have accomplished, you can always get better. There is no such thing as the perfect testing organization, or a perfect way to use data, which means there are always opportunities to grow and to get better. With that in mind, I want to present various skills and disciplines that will allow your program to grow no matter where you are at. If you have just started out, been testing for years, or are a massive optimization organization, there are always new ways to think and new skills to allow you to get more value out of each and every action that you take.

All programs are built off of key fundamental disciplines that allow all other disciplines to grow and to be acted on accordingly. All groups start out by needing to grow past the initial challenges and limitations that face your program. Most groups find a point where testing has shown a few wins, you find places to test where it makes sense, but you haven’t been able to get the level of consistency or buying that you are looking for. You are trapped in a world of explaining why you want to run a test, or working with different groups who aren’t interested in what the rest of the organization is doing. Early development programs have many factors that unite them; you might have been testing for 3 days or 3 years, and yet if you haven’t really taken the time to grow and incorporate testing beyond a superficial way, you will never achieve the true value that testing can provide for your organization.

The first thing that differentiates programs is the organizational pieces that have been put in place to help them move at a speed that will truly impact the bottom line. With that in mind, here are 5 key disciplines that will allow your program to the next level. For any testing program to work long term and to grow, you must first address the organizational roadblocks that will without fail trip up your program. The first and sometimes most vital aspect of growing your program can be adoption and the realization that this is not a once in a while type of action that you take.

Single Success Metric –

Without fail, the single greatest determining factor in the overall success of your testing program is have you unified on what you are trying to achieve and are you optimizing for the end behavior you want. So many programs fail when they try to optimize for the next page or a small action, or even worse to try and optimize for multiple metrics to make everyone happy. So many programs are less the sum of their parts, and because of this they are never able to impact the fundamental bottom line of the business at the level that they are capable of.

It isn’t just about optimizing for the goal of the site and not focusing on a individual concept, you must be able to answer all questions of this format: If A goes up, and B goes down, what do we do? What about A and C? Ultimately, you will quickly discover there is one thing that you really have to go up to be successful. It isn’t about clicks, our bounce rate, or just moving people to another part of your site, it is about focusing on what all of those behaviors are trying to accomplish. In just about all cases, the metric you will end up using is directly tied to how you make money, something like RPV, or lead conversion rate, or an engagement metric. What is important is that is global across your entire site, not just the section you think you will impact. That is what you need to focus on. You need to have everyone in agreement on how to act on the test before it launches, so that you don’t lose momentum or get lost in the internal quicksand of politics.

Nothing allows your program to be more valuable and for you to bring people together then to have everyone pointed towards the same end goal. Nothing ends internal debates faster than not worrying about clicks or sign-ups or search versus organic then optimizing for revenue and giving all actions an equal footing.

When I come in to assist groups and make sure they are getting the most value, nothing is more vital and I will refuse to go forward unless we can get agreement on what defines success and how we are going to measure it. You are often defined by what you don’t do as much as what you choose to do, and this is the seminal moment when you can take a giant leap in your program. This is the moment when a new discipline, unique to optimization, comes into place. It also allows us to treat any idea democratically and let the numbers define success.

Consistent Resources –

One of the many failings of early programs it he lack of ownership of the process and tool. Does IT, marketing, product, merchandising own testing? Whose resources get assigned and who will run this specific test? Do you have a FTE, or just a few random hours assigned to someone who has 10 other tasks on their plate? It is vital that as you build your program, that you have consistent time and resources assigned, outside of any specific test or project. The skills of testing, not just the tool, but understanding how your site works, how to move things, testing disciplines and how to navigate internal politics, all of these take experience and time. Programs that have new people jumping on board, people who run a test once or twice a quarter, or who have not clearly identified who will set-up, run, and do reporting on tests have the hardest time creating the momentum and cadence that allow your program to really blossom and to help insure that you are getting the magnitudes of value that testing can provide.

A good first step for any program is to assign a set number of hours per week. Assume that you will always be testing, and such, this won’t be tied to any individual test. As a good next step, you can make sure that any test you run can be done with that amount of resources in one week. Assigning the same people, a set amount of time, and making sure that you use it to always be moving forward will help your program achieve the cadence and scope of impact that you are looking for.

Infrastructure –

Having a proper infrastructure on your site is vital to be able to have a real program. The pieces of your infrastructure has two main components, the site code and the organizational components. A good robust infrastructure on your site should allow you to get a new test live within a day on any important part of your site without IT involvement. If you are creating testing code for each test, or if you require a separate ticket and full QA process for each test, you will never be able to move at the speed that a successful program runs at.

The other component for infrastructure is the organizational components. Do the important groups in your organization understand how you will run a test? Why you run a test? What will define success and how best to think about testing to provide meaningful answers? Are they bought in to testing or do they see it as something that threatens their imminent domain? Whereas the technical infrastructure is something that may be done mostly at the start and then revisited from time to time, the educating and working with various groups in your company is an ongoing thing and is vital for long term growth.

Rules of Action –

Once you have figured out your single success metric, do you have a consistent set of rules to define how you call a winner? How about when you have a winner, how does it go live, who is responsible, what is the timeframe? How do you build off of a test result? It is vital that you answer these questions and get people in agreement BEFORE you launch a test. Success and failure of any test is not in the lift it provides, but your ability to act on that data in a clear and expedited manner.

Groups will often write these down in a document and get signoff from different leads of the organization, or create a wiki and provide access to everyone. Each group has their own unique version of this, but it is vital that before you get to the action phase, that everyone is clear on how and why you do what you do. Nothing kills a program faster than people desperately trying to stop a change because they were “wrong” or people not willing to act on data because they were not ready to go. Never start a test if you are not able to act on the results, and rules of action are your way to ensure that you will be able to do this.

Program Champion –

Outside of having a single success metric, there may be no greater determination of long term success than having someone with some power in your organization who sponsors and pushes testing. Having a person who can navigate the political waters to enable the disciplines listed above, and to help communicate and hold people accountable for results. This person may not be involved in day to day actions, but they will often hold regular review meetings, talk to different groups, and present results to senior leadership. It is not about “owning” testing, optimization is part of everyone’s job, but this person helps get testing the attention it needs and holds people accountable.

You can look at any of the best testing organizations, and without fail you can name this person instantly and clearly. That person has bought in and while the program is often not perfect, they have helped lead it in a way that there is no doubt who sponsors it, why they push it, and how much value it brings to your company.

These disciplines represent just the first steps to really becomes a optimization leader, but they are vital in order for you to be able to grow and to facilitate the adoption of more advances skills. In the second set of disciplines, we will explore ways to think about testing to insure that you are getting more value from each of the actions which you just helped facilitate.

To navigate the entire testing series:
Testing 101 / Testing 202 / Testing 303 – Part 1 / Testing 303 – Part 2