As personalization and testing continues to become more and more mainstream you are starting to see a whole slew of groups that are being introduced to testing, or who may believe they have more functional optimization knowledge then reality. So many groups would get be better off if they just avoided a number of common pitfalls that befall new programs. While earlier I put together my list of the top 7 deadly sins of testing (no single success metric, weak leadership, failure to focus on efficiency, testing only what you want, confusing rate and value, and falling for the graveyard of knowledge), I want to instead give you a very quick checklist to make sure that at least your first few efforts in testing are not more painful and far more fruitful then what normally happens.
It does not matter if you have tested before, or are new to testing. What matters instead is how you tackle today’s issues and how you set yourself up to succeed. Breaking down the components of a successful first test allows you see the action items, and allows you to move towards the moments that really make or break a program. Nothing makes everyone’s life harder than starting out testing on the wrong foot, sine everyone will think that is how things are going to happen from then on. With that in mind, if you do nothing but follow this simple checklist, you are going to be far better off.
1) Decide on what you are trying to accomplish – Pick one metric for all your tests, across your entire site or org that defines success. This might be easy or hard, but it is vital. This is especially difficult for people coming from an analytics background who are used to reporting on multiple things and finding stories.
2) Pick one or two pages to start – Do not try to eat the entire elephant in one sitting.
3) Do not focus on your test ideas – You are going to have test ideas in mind, and you are going to want to only talk about and focus resources on that one test. Without fail this is where most groups want to focus, but I cannot stress enough how not important your concept for improvement will be to a successful program.
4) Make sure your first test is very simple – Don’t start trying to do a full redesign of your entire site or completely change a user flow. Pick one page and one thing to start. If you are not sure, then pick a page and design your first test to measure the relative value of all the items on the page.
5) Decide on your initial segments – Make sure all tests have a segment list. Not being focus on a specific segment will make it far easier to find exploitable segments and will start the process of learning even if you do not intend to use them right away.
Here are some basic rules for segments to make your life easier:
• Must be at least 5% of your total population (10% of a smaller site). This is total, not identified traffic
• Must have a comparable measure (you can’t measure new users versus Google users, since there are new Google users).
• Have to be defined PRIOR to the start of any campaign
• Need to cover the 4 big user information types (the listed items are just examples):
Day of week
Operating system (basically all the stuff you get from the user agent string)
How did the user get to the site
Key word types
Used internal search before
6) Start building a proper infrastructure – Build out a technical frameworkfor your entire site for testing. This may not be tied to the first test, thought that will be part of it. Get access so that you can run tests on 80% of your pages, using a combination of local and global set-up. A little pain up front will save you from lots of pain later on. It is always best to avoid death by a thousand cuts whenever possible, even if you don’t see that issue immediately.
7) Decide on rules of action – Make sure everyone is very clear on how you are going to call a test and act on winners before you launch your first test.
8) Making sure you are not going to QA tests like you do other parts of your site – So many testing programs are destroyed by letting IT decide on the work flow and on how you are going to QA your tests. You may have to work with your IT group, but it is vital that testing is not owned by your IT group but instead your marketing and product teams.
9) Point your resources towards feasible alternatives, not adding more metrics – Any additional time and additional resources need to be used on the following two things:
A) Adding as many alternatives to the test as possible, traffic permitting
B) Educating and working with groups to make sure they understand why you are testing and how you are going to act.
10) Remember that the most important time for your program is not the first test – Your first test is fun to focus on, but the period after your first test is where you can start to really focus on testing discipline. This is the time that defines whether you get good at just running tests, or if you build a rock star optimization program. So many groups fail because they miss how vital this time is.
There you go, 10 simple steps to make sure that your first moments in testing do not take your program astray. This is hardly the end of the road, but if you simply avoid setting yourself up for failure, then you can really start to look ahead to all the opportunity that is out there.
Most people are familiar with the famous Bertrand Russell Quote, “The whole problem with the world is that fools and fanatics are always so certain of themselves, and wiser people so full of doubts.” The challenge is how would you understand where you are in that paradigm? Are you the fool or are you the wise? How do you even know your real level of competence at any one moment? What is the difference between an expert and the average person?
One of the most famous psychological studies in the last few years is the Dunning-Kruger Effect. Its best description is:
The Dunning–Kruger effect is a cognitive bias in which unskilled people make poor decisions and reach erroneous conclusions, but their incompetence denies them the metacognitive ability to recognize their mistakes. The unskilled therefore suffer from illusory superiority, rating their ability as above average, much higher than it actually is, while the highly skilled underrate their own abilities, suffering from illusory inferiority. To put simply, when you don’t know what you don’t know, you have no ability to differentiate what is right or wrong.
Confidence is a tricky thing, as you need it to be able to stand up to challengers, but at the same time you need to be careful to not inflate it based on impression and not reality. Without confidence, we would never be able to convince others of any point we are trying to make. We have all dealt with people who obviously talked much more about their impact then could possibly be based on reality, but how do we know that we are not repeating the same mistake. Even worse, how do we know when others are playing on this psychological trick to take advantage of us, even if they do not consciously know they are doing it at the time?
The truth is that you will find far fewer real experts then those that claim to be. Statistically, an expert would be in the upper 5 or 10% of a certain field, yet we both have no way of measuring this and we are over run with experts claiming to be the best at what they do. Everyone thinks that what they are doing is the best way, otherwise they wouldn’t be doing it. Even worse, we get caught up in all sorts of promises, be it other customers claims, or their own skill set to evaluate that information. As the world becomes more complex, or as people from outside disciplines attempt to take their prior knowledge and apply it to a new field, they become even more susceptible to this problem. Even worse, like all biases, this impacts the more intelligent people more then the less intelligent. Dunning-Kruger is a double edged sword, as those that are most likely to be susceptible to the claims of experts are those that are least skilled in their own right:
“The skills needed to produce logically sound arguments, for instance, are the same skills that are necessary to recognize when a logically sound argument has been made. Thus, if people lack the skills to produce correct answers, they are also cursed with an inability to know when their answers, or anyone else’s, are right or wrong. They cannot recognize their responses as mistaken, or other people’s responses as superior to their own.”
This entire phenomenon is what causes the vicious cycles and what explains the over saturation in the analytics communities of people propagating the same tired actions by giving them new names and by finding others to make them feel good about their failed actions. Stories become the ultimate shortcut to show how amazing something is, without ever actually providing logical evidence to arrive at that conclusion. The truth is that people are rewarded for their ability to give people who already hold or held power ways to continue to run their empire, often with little relevance to getting results or to doing the right thing. This is certainly not unique to analytics, but it is important to this audience. We follow the greatest speakers, not the greatest thinkers. We worry about the best ways to make a presentation or to get reports out, not trying to stop entire conversations that are both negative to the organization and inefficient. We focus on how do we get others to see things our way, not on if we are seeing things the right way? Far more time is spent on how to convince others then it is on trying to analyze our own actions. One of the best quotes I have heard is, “There is no correlation between being a great speaker and having great ideas.”
So in the end, we are responsible to ourselves to look in the mirror and ask if we are suffering from a belief in others, or if we are discovering the best answers. Is convincing others and yourself that you are expert the most important action you can take? Just because you can make an analysis and use it to convince others, does that actually make it correct or valuable? Or is the discovery of the next right answer more important then getting credit for owning an action? It is up to everyone to decide what they really want to accomplish in their time, but if you are more interested in doing the right thing, then you must always be aware of Dunning-Kruger.
In order to do this, we must first set rules that help us hold ourselves and others accountable for what they do, in order to remove as much bias from evaluating success as possible.
Here are some simple steps to help make sure you are reaching the levels of success that you might believe that you are achieving:
1) Always ask, “In what ways can we challenge what we are doing?” or “How can I break this process”? No gain comes from doing things the exact same way you have been doing them.
2) Read, grow, look beyond your group. Know that you have never found the right answer, and the search is more important than the actual answer.
3) Define success up front. This is not just the goals your boss sets for you, but more importantly what it is that will define a successful program?
4) Make sure you are not measuring the outcome, but your influence on the outcome.
5) Seek out those that will challenge everything you believe. You do not need to agree, but only talking to like minded people is the fastest way to become the observed with Dunning – Kruger.
6) Assume that if you have not found a way to break a process in the last year or two, that you are not trying hard enough
7) Challenge everyone to take an idea to the next level. The first thing we come up with is comfortable. The next is growth.
8) Know that you will get an outcome from any action, so measuring just that does not tell you anything about the value you bring
9) If the words “I don’t know” are the end of the conversation for you, then you can be sure you are the sufferer of this bias.
10) Most importantly, change all the rules, and challenge all the rules, not to be difficult, but because you only get better by making others around you better.
These may seem like abstract general concepts and not directly related to your business or your day to day job, but the reality is that these are the actions that should define success there far more then the outside world. Growth is the goal, not the status quo, and as such we need to make change and going out of our comfort zones the priority, not re-wording past actions as new in order to convince others or yourself you have changed. Take others past their comfort zone and they will take you past your own. Keep getting better, and always know that you are never done and that you do not have the “correct” answer. Keep searching, and always question those around you, and you will always be vigilant against falling into the wrong end of Dunning – Kruger.
Testing can be the ultimate expression of this, you are free to test things far past your current comfort zone. You are free to not validate tired ideas but to explore and discover in a rational and predetermined way the actual value of things, not just the perceived value. In order to do this though, you must fundamentally want and prepare to discover these things. The greatest problem with most test programs is they never enable themselves to find out they are wrong, but instead focus on proving someone right.
The nice part is that just because you or someone you know suffers from Dunning – Kruger, it does not mean they always will. Every person you meet thinks they are doing the right thing, even when they are not. Change the conversation to the end goal, and talk about all the options that are in front of you, and you get past the egos that keep conversations from truly moving forward. Take the time to talk and to challenge people, and do not trust anyone that does not challenge you. You have many impartial tools that allow you to measure things and to work with others, but these tools only work when we use them in an unbiased manner, not to tell us what we want to hear.
One of the great frustrations of an optimization group can be working with other groups to build out and optimize their current initiatives. While the natural outcome of this frustration is shy away and just to hand all control of testing to each group, result show that this is consistently the worst way to get value from your optimization efforts. To add to this some groups think the only way they can expand is to add resources, instead of improving how and when you do certain actions. In a lot of cases, the most important battles that you end up waging having nothing to do with getting a test live or with sharing results, but instead in helping grow a new discipline throughout an organization. So how then do you work within your organization to build out the correct disciplines while making sure that you are able to test as often and as much as possible?
While there is no single answer to solving this ageless riddle, there are a number of common factors that differentiate the groups that do get real value, from the ones that have to come up with stories to justify their existence. These can often be the hardest, slowest, and most political actions you take for your program, but in the end they are almost always the efforts that produce the greatest returns. The practice of optimization is about change, not just in elements in a user experience, but also in the quest to improve how the organization itself thinks about and tackles problems. How these specific disciplines play out will always take a unique aspect to match your current organization, but the core disciplines will always remain.
DO – Make education a primary goal of your entire program
From the start, be it single success metric or just why and how you test, you have to take control and help others understand how and why you are going to tackle problems. Stopping people from measuring the wrong things can be more important then what you do measure, but also requires you to stand up and have the conversation that the person asking doesn’t want to have prior to starting the test.
Testing seems easy, but the reality is when you are doing it right it can cause a lot of confusion and discomfort for a lot of different members of each team. You are directly measuring that outcome of things and adding accountability to opinions that makes almost everyone uncomfortable. You need to be cognizant of their worries and current focus while helping them understand why you need to do things. It is vital these conversations happen before you take any action, otherwise you will inevitably either sub optimize or even worse, get into a political battle between groups.
DON’T – Try to tackle the entire organization at one time
There is a famous saying, “How do you eat an elephant? One bite at a time”. The same holds true for growing your practice within the organization. Don’t try to convince everyone to do the right thing, instead focus on specific people or groups, and start with less high profile ones. No one is going to just change for the sake of change, especially since almost all change you are looking to create is against their own personal empire and agenda. Once you show success with those smaller groups, and you show how discipline played a big role in the outcomes, this will help you convince or deal with other groups and eventually you will find that the entire elephant has been eaten.
DO – Make sure that all testing conversations focus on a series, not on the execution of an idea
People always start testing thinking of ways to validate their single idea. They think X is better than Y, and want to use testing to prove it. This is without fail a massively flawed and inefficient manner to test, yet it is where a lot of groups stay due to the ease of just doing what others want. The key to any test is to make the learning, growth, and iterative improvement part of every conversation, starting at day one. Never let conversations just stand that are about which is better, but instead change them to the discovery of what is valuable and how do you exploit that information. It may cause a bit of discomfort at the start, but the reality is that if you do not make this a priority, you will inevitably start getting into the rut of single concept tests.
DON’T – Report everything that everyone wants
This is especially uncomfortable for analytics groups that try to add testing on as just a function. Part of optimization is the discipline to focus on what does matter, and to not pretend that you can answer the whole slew of questions that come. People bought more, but what did they click on? Where did they go? Why did they do what they did? The reality is that you have no clue even the correlation from a single data point, so that data is irrelevant. People spent more, but what product? Again, this will only cause conflict between specific product teams and do nothing to focus on the change that needs to happen.
The key is to focus on education and your single success metric to understand why you don’t do these things. It is ok to look at other metrics, so long as you do not draw ANY conclusions from them in a single test, and you look at ones that have a chance to provide value (this means not clicks). You will be able to spot patterns only after you collect a large number of data points and are disciplined in how you look at the larger picture of your optimization efforts.
DO – Make the analysis of optimization patterns far more important than a single test
Over time you will start to build out knowledge across tests. While it is easy to focus on only what you are dealing with in any given day, the reality is that the real lessons that you can learn don’t come from a single data point, but across large numbers of tests. These lessons, be it what types of changes or where you should focus, become the most valuable piece as they shape your entire future product road map. This means that you need to make this information available, but also champion these larger lessons and help others understand that specific tests probably won’t reveal as great of lessons.
DON’T – Just run an idea someone brings to you
Deconstruction, the ability to look at the assumptions that made someone come to a conclusion, becomes a vital skill in the hands of a great optimizer. Every time you tackle an assumption, you enable a greater possible outcome and also allow each test to possibly impact far more then intended. Want to test content to a specific user group on your front door? Does content matter most? How about which group? How about where in the user flow? You can easily start building tests to have enough variations and to tackle these larger questions, but only if you make it a priority. One of the great complaints of optimization is the concept of local maximum, but the reality is that this mostly comes from the limits of the imagination of the testers, and not from the specifics of the test itself.
I am sure that most people at multiple points here asked how realistic these suggestions are? There are hundreds of excuses why people do things, but the reality is that it is up to you to stand up and do the right thing, even if uncomfortable, at all times, otherwise you will find it nearly impossible to do the right thing when it is required. If you are proactive in your education, and make these a focus, you will find it is far easier to push back when and as needed.
There are hundreds of other possible things you can do to expand the optimization practices throughout your organization. If you simply nail these few items, you will avoid almost all of the major pitfalls that lead to programs becoming stagnant or pretending they are generating far more value than they really are. No step you take will be easy, but if you are consistent, up front, and make these actions a priority over the specific actions of running a test, you will be amazed at how far the entire practice has gone in just a few months.
There is a lot of misunderstanding on the difficulty of using Adobe Target. There are all sorts of uses of the tool, and in a lot of cases, people confuse their inability to set-up a proper infrastructure with limitations of the tool. The wonderful thing about Target is that it has 5000 different uses. The difficult thing about Target is that it has 5000 different uses. Figuring out what to do, when to use it, and how to tackle that problem is paramount to a mature program.
Before we get to specific suggestion however, I want to make it clear what I consider a proper technical infrastructure. It comes down to how you answer this question? Can you get 80% of all tests live, with the proper amount of variations, live in 30 min or less. That is from concept to launch. That number seems crazy to people at first, but the reality is that you can get a test fully live in 5 min if you have everything ready to go. That is the sign of a successful infrastructure: speed of execution, understanding and flexibility. In order to do that, you have to have things set-up in a way that will make you successful. Here are the key things to do and avoid in that quest for speed and execution without sacrificing (in fact improving) results.
Do – Make sure that you can test on all major part of your site today
Make sure that you have mboxes on all key parts of your site. You will most likely want to make sure that you have multiple mboxes on those key pages, usually leveraging a mixture of global and wrapped mboxes throughout the page. Global mboxes are ones that do not wrap any content, but exist at the very top of the page to control CSS, redirects, and other functions. Wrapped mboxes are ones that wrap key elements of the page, with the best practice of never wrapping more then 1/3rd of the viewable page with a single mbox. The key here is to make sure you are not adding and removing mboxes for each test. One of the nice things is that even if you do have those mboxes on the page, you can disable them easily in the console so that you are not charged for them.
Don’t – Have more then 5-6 mboxes on any page
Just because you want to prepare a page to succeed does not meaning going crazy with mboxes. The reality is that each request does impact site performance, and any noticeable interaction with the page will result in the validity of a test result being highly questionable. Setting up a page to be able to do 80% of the things you can do is important, just as prioritizing those efforts to leverage what you can do today to learn about the other 20% that require additional resources. If you can do 80% of possible things today, then do those things, and you will find that often the tests you are not sure about are the ones that produce the biggest and most meaningful winners.
Do – Make sure that you are tracking key information about your users
This is both an on-page and in console element and may require a bit more time and discipline to think about before you act on anything. This means leveraging one of the most under-utilized but extremely valuable parts of the tool in the profile system. Make sure that you are passing key information you might have from your CMS or other systems about your users where necessary, and also make sure that you are recording key pieces of information in the profile system. The key here is to look into it, but not believe it blindly, to search for value and not predetermine outcomes. Just because you really want to target to a certain membership level does not mean you should ever target to them before you test and measure the value of that change compared to other alternatives. Even if you aren’t leveraging it today, it can be leveraged in the future, and it allows you to easily add that information for analysis later.
Don’t – Delay your infrastructure for unnecessary pieces of information
Most groups tackle optimization as just the end function of a number of different inputs. The worst examples of this is when groups go through and interview for a list of all possible pieces of data a group may want to look at or segment information by. While it might make your life easier since you don’t have to correct anyone, the reality is that you are most likely causing massive damage to your program. It is a good idea to get a read on what people may want, but the primary directive should be to help groups think and act differently with data. As part of that, you need to ask the fundamental question of “will the data I am collecting improve our business enough to be worth it, and even so, can I get better outcomes from using those same resources?” There will be cases when the answer is yes, but in most cases if you are honest the answer will be a resounding no. In those cases, do not delay other actions just to make others happy. Use what you have to get the most with what you got, don’t waste time waiting for everything to be perfect.
Do – Make sure that you can get tests live without IT involvement.
You should be able to do 80% of tests with some very basic understanding of how a webpage works. After you get your infrastructure in place, you do not need IT support except in one off cases. You should not need IT except I extreme circumstances and only then once you have proven via discovery that the item you are using those resources on is the most efficient use of everyone’s time. Where groups run into major problems is when they view each test as a technical project, which gives both the wrong impression about the difficulty of testing and makes things much slower to react then they should be. Even worse are groups that think that technically complex means valuable, which is almost always the opposite of the truth. Difficult means difficult, valuable means valuable, those two things do not have much to do with the other.
Prioritize tests by how much effort they are to get live, as well as how many different feasible alternatives you will get, and how much they challenge assumptions. Do that, and you will dramatically improve the outcomes and reduce the resources needed for your program.
Don’t – QA testing like you do everything else on your site
There is nothing worse for testing then having a test ready to go and then having to wait weeks for it to go through a standard QA process. Make sure you do a good job of QA, but keep in mind that nothing will ever be perfect, and that you can get most QA done for most tests by simply passing around a few QA links to a couple of colleagues and having everyone use a couple of browsers. You will most likely want to do more QA for efforts that dramatically change site function, but the reality is that you should be fewer of those tests and far more of the more basic tests anyways. Add better rights control so that fewer people can then push things fully live also adds meaningful limits to make sure you are accomplishing what is needed without sacrificing speed and efficiency.
The promise of the tool is that marketers start having control of their site, so why then do people so willingly give up that control? You need to be able to understand why you need to QA differently, and how different types of tests impact site function, so that you can have a meaningful conversation with yoru tech team and help them understand that you are not going to be constantly breaking their site. Be thorough, but don’t go crazy or just dump the responsibility onto others.
Do – Make sure that everyone is aware of when you are running a test
One of the benefits of setting up your campaigns to use QA parameters is that those links can be shared with everyone and you can therefor make more people aware of a campaign. There is nothing worse than having a campaign live for a few days and then having to stop it because some VP randomly hit a test variant and thinks the site is broken. Communicating both launch and results and especially lessons learned across tests is also part of a successful infrastructure.
This will also help build awareness and interest in the results of the tests. If you design tests with enough variants and challenge enough assumptions, in almost all cases you will get results that make other groups fundamentally challenge their own ideas of what works.
Don’t – Ever launch a test without a clear idea of how you are going to act on the data
There is no point in any test if you are not clear on how you are going to act. This means knowing your single success metric, but also what happens to push a winner, or how you are going to follow-up, or what you will do with segment information. Having those conversations outside of the specifics of a test and having groups aware of their roles before the launch is vital to a speedy and successful infrastructure.
Being prepared to act and having your groups ready to act quickly and efficiently can many times be against the entire history of some organizations. This is why it becomes so vital to build a proper infrastructure on every level to enable testing to work. Focusing on what matters, instead of just how to get a single test live, is one of the key differences between groups who just test and those that run successful testing programs. Make sure that you focus on the things outside of a specific test and help move the various groups involved towards an end goal, and you will be amazed at how fast you will act and the results you will achieve.