One Problem but Many Voices – The One Thing People Need to Understand about Optimization

The hardest challenge when working with different groups in the optimization space is often trying to get past their misconceptions and to help them view optimization in a different form. It doesn’t matter if they have been doing testing for 1 day or 10 years, there is still a massive difference in efficiency and the value that can be generated. Results are not random yet so many believe they are because they misunderstand optimization on the most fundamental levels. The reality of real successful optimization is often far from the perceived reality from those just entering the space. The number of misconceptions is so large that it can often be nearly impossible to prioritize them or to tackle them all.

Because this problem is so common, I reached out to the smartest people I know in the industry and asked them to share their thoughts about what the one thing they wished people understood about optimization.

Rhett Norton – Consultant

One thing that I wish people understood about successful optimization is that testing is about discipline. To truly be successful you need discipline in how to think about testing, how to take action, how to organize internally, how to learn iteratively, how to communicate results, how to learn what influences segments, how to build a program, and how to create a culture. It isn’t about launching tests or how many tests you run. It isn’t about creating really big tests. It isn’t about personalization. It isn’t about moving your political agenda forward.

Without discipline companies go through the motions of testing without ever really achieving amazing long term results. The most successful companies I’ve worked with have been successful with creating discipline in parts of their testing program. I’ve never seen a company that is disciplined in every aspect of optimization, but hey, maybe your company could be the first.

Drew Phillips – Consultant

I wish that more people understood that optimization is a disciplined, yet free form process. It is disciplined in that you can’t be successful by simply throwing the spaghetti at the wall. Testing random ideas will get you nowhere fast.

It is free form in that you need to have the flexibility to optimize elements that you find to be influential, not lock yourself into a specific roadmap. Optimization is a process that changes as you learn from each campaign. You will get the most out of your optimization efforts by iterating off of things you learn from previous tests.

Brandon Anderson – Consultant

The one thing I wish optimization practitioners understood is the 80/20 rule and the need for focusing on the “basics”. 80% of optimization ROI comes from doing 20% of optimization activities. The optimization umbrella is getting bigger and bigger – web, mobile web, mobile app, email, display ad – and the number of activities in these areas is almost infinite – banners, images, copy, buttons, layout, color, page flow, etc. It’s very easy to get excited about new initiatives like personalization and omnichannel. These things may have value. But is their value greater than the “basic” activity of optimizing page layout in the checkout funnel?

Sometimes organizations that have been doing A/B testing for years feel like they need to work on complex activities in order to continue progressing. My experience is that even mature organizations need to look past the hype of new and shiny buzzwords and determine which activities will give them the highest efficiency. Get the 80% with 20% of the effort by focusing on the basics.

Ryan Roberts – Solution Architect

I wish more people realized that successful optimization has to be a process that will require time, effort and thoughtful strategy. Just throwing together some random tests misses the point and the benefit of a well-run optimization program.

I also wish people were more careful about how they read test results. People that rely solely on confidence calculations are going to end up with a lot more wrong conclusions than they think. They need to understand what the rules of conclusive results should be for their site. And they have to apply them religiously to each test they run.

Doug Mumford – Consultant

Many great tests don’t (or shouldn’t) take much development time to setup. Orgs should actively work to reduce lead time from idea to launch. Launching a test in under an hour is very possible. Orgs tend to anchor their perception of development time based on what they’ve done in the past – 4-8 hours for dev slated out two weeks in advance, 3 hours for QA. Why?

While there are some tests that will require more time a lot of highly valuable tests can be done with three lines of CSS or jQuery, loaded up in four browsers to make sure everything looks good (and perhaps an iPhone and iPad), and launch. Have a bias for action.

If I had to characterize my own answer to the question it would be that there is a massive difference between action and value. Just running a test, be it one or 500, is not the mark that you are successfully optimizing. Optimization is about how you tackle large assumptions, and about how you act on data, and even how you think about what data can and can’t tell you. So much time is wasted in the pursuit of executing on assumptions and against the propagation of agendas which is the exact opposite of where the value of optimization comes from.

It is about discipline, and statistics, and variance, and technical solutions, and dealing with senior management and dealing with biases and assumptions. It is all that and more. It is a means to an ends, but that end is increased revenue for your organization, not just blindly reaching an audience or making an individual look good. The more you try to justify a specific action or the more complicated you make something, the less value you get and the more time you waste. Just understanding that action in and of itself is not the answer is the first step to being truly open to solving the largest challenges that optimization programs face. The challenge is never in running tests, the real challenge is finding solutions and ways to even have these conversations.

What do you find as the one thing you wished people understood about optimization? What are you doing to solve it?

When Heuristics go Bad – Dealing with Common Optimization Practices – Part 2

My first trip through the common heuristics of conversion rate optimization looked at two of the more common testing ideas and how they usually reach false or limiting conclusions. In my second part I want to look at general testing theory best practices and how they can be major limiting factors in the success of your program.

It is important to remember that you are always going to get an outcome so this is not about can you make money. How you and the people in your organization think about testing is the largest factor in what you value that optimization produces. This is an evaluation of the efficiency of the method and how much does it produce for the same or less resources. In concept you can spend infinite amount of resources to achieve any end goal, but the reality is that we are always faced with a finite amount of time and population, which means we must always be looking for ways to improve inefficient systems. If we continue to be limited by these common heuristics then the industry as a whole will continue to produce minimal results compared to what it can and should be producing.

Always have a Hypothesis –

There is not more misunderstood term then hypothesis. In all likelihood it is because most are familiar only with their 6th grade (at least in my school) science instruction or they took classroom formal science in college. In those fields we operate like we have unlimited time and resources and we are trying to validate whether a drug will cause cancer, not whether a banner will get more clicks if it is blue or red. The stakes are higher and the models are much more simple in classroom controlled studies for cancer. There is a lot to scientific method, especially when approached from a resource efficiency perspective that is not considered in such a simplistic view of idea validation.

We must apply scientific rigor, but we must also make sure that all actions make sense in real world situations, which means that efficiency and minimizing regret are more important than validation of an individual’s opinion. It is not that scientific method relies on the use of a hypothesis, it is simply that we mistake a hypothesis with a correct hypothesis; we seek validation for our opinions and not the discovery of the best way to proceed. Science is also about proving one idea versus all other alternative hypothesis yet we ignore that part of the discipline because it is not the part that allows someone to see if they are right. In the grand scheme of things we are drastically over valuing test ideas and that is distracting from the parts of the process that provide value.

Let’s start with the basics. You should never, and I mean never, run a test if you do not have a single success metric for your entire site. In most cases this is to make more money, but whatever it is, this goal exists outside of the concept of the test. You must also must have rigid measurement and action rules that are reproducible, which means that you must understand real world situations like the limitations of confidence and variance.

You can then have an opinion about what you think will happen when you make a change. The problem is when we confuse that opinion with the measured goals of the test. Even worse we limit what we compare resulting in massively inefficient use of your time and effort. Just because you believe that improving your navigation will get people to spend more time on your site, that is completely irrelevant to the end goal of making more money. Your belief that more engagement will result in more revenue is not enough to make it so. If you are right AND if that also produces more revenue, then you will know that from revenue. If you are wrong you will only know that from revenue. We must construct our actions to produce answers to our opinion and to what is best for our organization. Hypothesis and ideas are just a very small part of a much more complex and important picture, and over focus on them allows people to avoid the responsibility and the benefit on focusing on all those other parts, which are the ones that really make a difference over time for any and all testing programs.

The worst factor of this is that it allows people to fall for congruence bias and to fail to ask the right questions. We become so used to the conversation around a single idea that the concept of discovery and challenging assumptions is more word then action. Questions can be incredibly important to the success of a program, but only if they are tackled in the right order and used to focus attention, not as the final validation of spent attention. If your hypothesis is that a certain navigation change will result in more engagement, then the correct use of your resources are either which of a number of different versions of the navigation will produce the most revenue or if you can, which section on your site produces the most engagement when changed. In both cases you have adapted your “hypothesis” to present a more efficient and functional use of your time. The hypothesis exists, but it is not the constraint of the test. If you are right, you will see it. If you are wrong, you will make more money.

This means that having a hypothesis is important, but only if it is not the test charter. Have an idea what you are trying to accomplish and make sure that you go about seeing the value of certain actions compared to each other is more important. Sometimes the most effective hypothesis are “I believe that we do not know the value of different sections on our pages.” Don’t confuse your opinion on what will win with a successful test. Challenge assumptions and design efforts to maximize what you can do with what you have and you will never be without opinions. The best answers are always when you are proven wrong, but if you get too caught up on validating your hypothesis, then you will always be missing the largest lessons you could be learning.

We need to optimize X because it is losing Y

This is the classic problem of confusing rate and value, or more correctly correlative and causal inference. We confuse what we want to happen with what is really happening. Just because people were doing X and now they are doing Y, it doesn’t mean that this is directly causing any change, positive or negative to our end goals. Outside of the three rules of proving causation the real issue here is that we get tied to our beliefs about a pattern of events even when the data cannot possibly validate that conclusion. Understanding and acting on what you know as opposed to what you want to have happen is the difference between being data driven and simply being data justified.

Think about it this way, I have 23% clicks on one section of my page and 0% on another. If I were to improve one of those which one is going to produce the biggest returns? The answer here is that you do not know. A rate of interaction cannot possibly tell you the value of changing that item. Some of the most important parts of any user experience are things that can’t even be clicked.

This plays out outside of clicks too. We have a product funnel and we see more people leaving on page 3, therefore we need to test on page 3. The reality is that more or less people may or may not be tied to more or less revenue. Even if it is tied it may be a qualification issue higher, or a user interaction issue, or simply too many people in a prior step. This is called a linear assumption fallacy, where we assume that when we have 5 people and 2 convert that if we have 10 people 4 will convert. Linear models are rare in nature but are easy to understand, so we fall back on comfort over realistic understanding.

The act of figuring out what to test can be difficult but it is never improved by pretending we have validation of our own ideas when we have nothing to justify them. We need to be open to discovering where we should go and to focus on some set path. In almost all cases you will find that you are wrong, often dramatically so, about where problems really are and how to fix them. This is why it is so important to not try and focus solely on more or less correlative actions. We can and should be able to test fast enough and with few enough resources that we will never be limited to this realm unless we can are stuck there mentally.

Like so much else what you spend your time and effort on is incredibly important. There are a thousand things you can improve and there are always new ideas. Justifying them falsely or focusing on them instead of the discipline of testing is nothing but a drag on your entire testing program. Test ideation is about 1% of the value derived from a test program yet it is 90%+ of where people like to spend their time. A 5% gain that took 2 months is worth a lot less than a 10% gain that took 2 weeks. The most important issues we must face are not about generating test ideas or validating our beliefs about how to improve our site, it is about discovering and applying resources to make sure that we are doing the 10% option and not the 5% option. If we overly focus on test ideas and not the discipline of applying them correctly we are never going to going to achieve what should be achieved. If we get lost trying to focus only on where we want to go, then you will always be limited in the possible outcomes you can generate.

When Heuristics go Bad – Dealing with Common Optimization Practices – Part 1

Talk to 5 people in the optimization space and you will get 5 different stories about how best to solve your website. Talk with 50 however and those 5 will get repeated more often than not. Such is the world we operate in where “best practices” become so common place and repeated that we often do not take the time to really think about or prove their effectiveness. Because of this phenomenon a lot of actions which are less than ideal or outright bad for companies become reinforced must do items.

The reality is that discipline is going to always win out over specific actions, and that often times the best answer is to measure everything against each other and take nothing for granted. While all of that is true it is still important you understand these common suggestions, where they work, how, why, and more importantly why people believe they are more valuable than they really may be.

Test Free Shipping or Price Changes

This is a real common one for retail sites as it is easy to understand, and a common tactic (thanks Amazon) and one that is easy to sell to the higher ups. The problem is not actually the concept, but how people measure the impact of it, and what that means to other similar tactics. What can easily seem like a huge win is often a massive loss, and even worse due to how most back-end systems are designed the actual amount of work needed to achieve these tests can be much higher than other more simple and extremely valuable uses of your finite resources.

Let’s look at the math of a basic free shipping test. In this simplified scenario, we sell 1 item for $90 dollars on our site, with an actual cost of $70 to us ($20 net profit). Our shipping is $10 dollars, which means that when it is normally purchased someone pays us $100.

We want to test free shipping, where we pay for the shipping and sell the same widget for now $90. We run the test and we have an 50% increase in sales! We should be getting promotions and in most cases the person who ran this project is shouting their accomplishments to the entire world and everyone that will listen. Obviously this is the greatest thing ever and everyone should be doing it… except you just lost a lot of money.

The problem here is that we often confused gross and net profit, especially because in a lot of different tests you are not directly changing the bottom line. In the case of free shipping or pricing tests though, we are directly change what a single sell means to us.

Let’s dive into the numbers of the above. Let’s say that we sell 1000 orders in our control normal group.

$100 X 1000 = $100000

But the real number that impacts the business is:

$20 x 1000 = $20000

In the free shipping option, we have cut our profit in half by paying for the $10 shipping, which means that at $10 profit we actually have to have twice as many orders JUST TO BREAK EVEN.

$20000 / $10 = 2000

This means that if we fall back to the standard RPV reporting that you look at for other types of tests, then the math says that:

$100 X 1000 = $100000
$90 X 2000 = $180000

So any option where we do not increase RPV by at least 180% means we are dramatically losing revenue. So many times you see reports of amazing results from these kinds of optimization efforts which are masking the realities behind the business. It can be hard, no matter how much this makes sense in conversation, to have the discipline to think about a 50% increase as a loss, but that is exactly what happened here. Sadly this hypothetical story plays out often in the real world, with the most likely result being the pushing of the results and not the rational evaluation of the impact to the business.

This same scenario plays out anytime we have varied margin and not as varied gross cost. The other common example is price changes, where the cost of the item remains fixed, but the test is only truly impacting how much margin we make off of the item. In both cases we are forced to set minimum marks prior to starting a test, and treating those as the neutral point, not the normal relative percentage lift that we might be accustomed to.

Always repeat content on your site

This and a large number of other common personalization type suggestions (who to target to and how to target to them) actually have a large number of issues inherent to them. The first is that even if what is suggested is true, it does not mean that it is the most valuable way to tackle the problem. Just because repeating content does improve performance by 3%, it doesn’t mean that doing something else completely will not result in a 10% or 50% increase.

The sad truth is that repeating content, when it does work, is often a very small incremental gain and pails in comparison to many other concepts of content that you could be trying. The goal is not to just do something that produces an outcome as every action produces an outcome, the goal is to find the action that produces the maximum outcome for the lowest amount of resources. In that light repeating content is often but not always a poor use of time and resources. The reason it is talked about is often not due to its performance but because it is easy to understand and easier to get buy-in from above.

The second major problem with these is that they skip the entire discipline that leads to the answer. There is no problem with repeating content as long as you also try 3-4 other completely different forms of content. Repeating content may be the right answer, it may be an ok answer, and it may be the worst answer, but you only know that if you are open to discovering the truth. There is no problem having a certain group or behavior you want to see if you can target to, the issue is when you target to them without looking at the other feasible alternatives. If you are not testing out multiple concepts to everyone and looking at them for the best combination, then no matter what you do you are losing revenue (and making you and your team do extra work).

The real irony of course is that if you test these out in a way to find out the impact compared to other alternatives, the absolutely worst case scenario is that you are correct and you target as you would have liked. Any other scenario presents you either with a piece of content or the group or both that results in better performance. Knowing this information allows you to save time and effort in the future as well as spend resources on actions that are more likely to produce a result.

It is not unusual to find that doing just targeting to a specific group will result in that group showing a slight increase, and if that is all that you look at you would have evidence to present and share internally as success. Looking at the issues deeper you commonly find that the overall impact to the business is negligible (within the standard 2% natural variance) or even worse negative to the whole. It is also not uncommon to find a combination that you never thought of presenting a massive gain.

One of my favorite stories in this line was when I worked with an organization that had decided exactly how and what to target to a number of specific groups based on a very complex statistical analysis of site behaviors. They had built out large amounts of infrastructure to facilitate this exact action. We instead took 100% of the same content they already had and presented it to everyone, looking at the impact to serving it to the groups they envisioned as well as others. We simple took all their existing content and serve it to everyone and also in a few different dynamic permutations. The result showed that if they had done only what they had envisioned they would have lost 18% total leads on the site (this is also a great example of why causal inference is so vital and to not rely on correlative inference). They also found that by serving 2 of their normal pieces of content based on behaviors they had not envisioned they would see a 20% gain. They were able to go from causing dramatic harm to their business to a large meaningful multimillion dollar gain simply by not relying solely on hearsay and instead testing their assumptions.

In both cases there are many different ways you can manipulate the data to look like there was a positive outcome while actually doing damage. In both cases massive amounts of time and effort was spent to try something only to find an outcome counter to people’s assumptions. In both cases testing out assumptions and exploring to discover the value of different actions prior would have better informed and created more value.

In the end, any idea is only going to be as valuable as the system you put it through. There is nothing inherently wrong with either concept as long as they are measured for efficiency and acted on rationally. If you can take a common heuristic and evaluate it properly, there is value to be had. That does not mean that they will act as magical panacea, nor should you plan your program around such flawed simple ideas. Focus on building the proper system and you will be able to provide value no matter what concepts get thrown your way.

The Harsh Realities of the Business World

I was recently lucky enough to get to take 6 weeks off for a sabbatical, where I was able to really get away from the life of a consultant and just spend time with my family. Upon returning I reached out to a number of people with whom I work or know in the industry and wanted to catch-up. I had two different people, both of whom I respect deeply and who I think are some of the best and brightest in the industry, regal stories about how fed up they were with the industry and how they were losing faith in the system. This conversation is hardly new, but the stark difference between the “real” world and what happens in the corporate world was striking.

Without fail everyone comes to realize the limitations of the corporate world, and while every organization is different, the things you see, especially at the largest corporations, are almost all universal. One of the hardest things I have to deal with as I try and mentor new people or people who I want to help in the industry is help them really come to grips with this reality and help them see that there is hope, but that they will always be making choices: what is good for the company vs. what is good for them.

With that in mind, I wanted to help express some universal truths that I think everyone should be comfortable with if they want to really exist for any period of time in this business world.


Most effort is wasted –
This becomes striking clear when you start doing exploratory casual analysis and look at the impact of work or entire departments. The number of times in the last 5 years that I have taken a few minutes of effort and the end result has shown that entire years’ worth of effort had negative impacts to the bottom line cannot be counted on my appendages. There are entire disciplines that people have devoted their entire life to that have no impact whatsoever and are nothing more than phrenology.

But 100% of people think they do excellent work – This really hit home today with one of the conversations as the realization that action is confused with value really came home. Most people assume their actions are providing value, and because of the preponderance of data out there, most can find ways to come up with some story to justify their actions.

I have had multiple engagements where it started with the person showing reports and graphs and presentations showing massive value to the program, only to take a few minutes and dive into the numbers and show that not only was it not improving the business, in multiple cases they were actually causing catastrophic harm to the business. It happens everywhere, at least 70% of case studies are full of 100% fake data. People are so desperate to please their boss or make themselves look good they find ways, often subconscious, to show their value. It is not malicious, it is just sociopathic.

My favorite story of this was when I was working with a group that was reporting how great their recommendation tool was and how it was generating 18% more revenue! In reality they were only looking at the revenue increase of the products recommended (3 out of a library in the hundreds). When doing just a standard analysis of the entire revenue stream, there were piles of data that showed they were losing 6% net revenue for the entire company, totaling millions and millions of dollars.

People are not rewarded if a company makes 3% more money from their actions, and as such they treat the lack of complaining as the ultimate sign of success. We never look at what could have been, only what was and how people reacted to it. Context is something you have to strive for and work hard to get, every day, otherwise almost all stories and data is meaningless. This often leads to many long term problems…

Because of this most people have no clue what they are talking about – If you can come up with a story to justify any action, you don’t dive to see if you are right, most effort is wasted, and the only thing people look at are the stories you weave, why would you need to know what you are talking about? It is far easier to create a narrative out of the air then it is to actually be able to back up anything you are talking about. Thanks to fear, Dunning-Kruger, and just common greed, this is allowed to take place. The more someone is able to convince people of their story (which is a different skill then actual results), the more they move forward, the more people believe it, and the more people want to copy it. It is a self-fulfilling cycle and one where actual knowledge is scorned because it serves as a direct challenge to the empires built by these people. If you really want to improve things, you must always make people go where they don’t want to, because the safe shores are the ones without accountability and that sound like the same things they have always been doing.

People who build the tools and work at agencies often know even less – I have direct experience with a large number of tools, and I am lucky enough to know “thought leaders” at a large number of other tools and agencies, and I can tell you as a whole most of the people you hear talking couldn’t provide value to you if their lives depended on it. They have become experts at telling you what you want to hear, not telling you what you need to hear. The top people in the industry are story tellers who weave a tail of telling you to do basically what you have been doing, but justify it with fancy terms or new actions to get to the same place. Tools become designed for this, people get advancements for this, and oftentimes anyone who doesn’t want to take part of this vicious cycle move on to other endeavors, meaning the worst become the ones their longest, gaining power and only making the cycle worse. I can attest that the top 5 agencies I know in my space, I know multiple top people at all of them and not a one knows anything about providing value, or even cares. They do the same tired actions because that is what they have always done, and they don’t get called out on it, so why should they change? To quote one famous person in our industry, “I throw a grenade and try to get people to come to me when they run from it.”

And you thank them for it – You know why the tools are designed to do things that aren’t valuable and the top agencies are run by people who tell stories and have no clue what they are doing?

Because you do not hold them accountable

I have worked with exactly 3 organizations in 12 years where results mattered, the rest just want to sell a story internally, do something new, and then do more actions that make their boss happy. You buy the story, do the failed actions, sell that story internally, which results in promotion which propagates the cycle. The cycle spreads and just as stated before, knowledge of other ways is simple a risk. This is why you find people in these places that are so good technically, but very few if any in most organizations that have any clue about strategy other than repeating the same tired failing things that everyone else repeats. Organizations want people to do what they say and tell them it is golden, not to make them money. The only person who is going to really hold you accountable for value derived is yourself.

But all hope is not lost – This environment is where we exist, and it has been that way since you started and will be far after you are done working. The environment doesn’t change, so it is up to you to decide how to deal with it. Just because people don’t want to change doesn’t mean they won’t, it just means it isn’t easy. Just because others don’t hold you accountable does not mean that you can’t. Just because doing what others want will help you move forward, it doesn’t mean that you have to sell out at all opportunities. It is a balancing act, both of survival and how to tackle these complex problems. The thing that makes people survive or be good is that they don’t hide from the reality, they embrace it, they might get frustrated, but they come back and push back even harder tomorrow. If you give in, if you become cynical, if you just give up and take the easy path, that is your decision, just as it is to do the right thing even if it is not best for your career. No one can tell you what the right choice is, all they can do is help you see that you are making these choices and help you make the one that is best for you.

And when you do overcome these problems it can be the ultimate high –
Just because the entire system might be designed to keep things from progressing does not mean that progress isn’t made, only that it is rare and incredibly hard fought. I had the pleasure to also be on a call today with a client who has come from an org with no background in testing, who just threw up tests because they thought they should and who had no resources and no knowledge, who in the last 9 months has transformed to the point that they have a separate team, great discipline, good educational base, and who is running a series of exploratory tests. That moment, which I wish happened all the time but doesn’t, makes it all worth it, at least for me.

In the end, it isn’t about what title you have, who thinks you did well, it is about what are you trying to accomplish and did you hold yourself accountable to it? People can do good work, almost all of the problems I outlined above happen subconsciously, not consciously. People aren’t out to screw each other; they just do it and then rationalize it away. Opening up someone’s eyes, or making it so they don’t have to do the easy thing just because is all you can do.

Choose what you want, and then do it. Don’t let the system dictate the outcome, it is up to you to overcome, adapt, or become a cog in that machine.