Rant – Door #1 or Door #2?

There is a moment in every engagement where you have to make the choice between making the client happy or making them successful. There is always something (and usually many many things) that they want to do, that will lead them down a very dangerous path. Those same things are almost always their babies, tied to things like personalization, or improving metrics that make them look good, or a hundred other things.

When that moment comes, it is amazing how often people choose the “make them happy” path. Most do it by not acknowledging the issues, other make excuses to themselves that they will deal with it later, or that the client was going to do it anyways. Most make sure they are in a state of plausible deniability by not knowing what the value of things are or by believing the so called “experts” of their field, most of which know less then a low level analyst, but who focus on telling people what they want to hear and by being good speakers (There is no correlation between being a great speaker and having great ideas”.

One of my favorite managers used to have a saying. Whenever you didn’t do what you were supposed to, he would ask you “door #1 or door #2?”. This was a reference to lets make a deal, where door #3 is the prize (you did the right thing), and behind the other two doors, you got the donkey. In this case, Door #1 refers to the fact that you are incompetent – You tried but weren’t up to the job. Door #2 refers to you giving me the middle finger. You could have tried more, or could have done more, but you chose not to.

When you make that choice, when you choose to improve your relationship, or you choose to make them happy, over making them successful, which door is it? What is so important that you can’t do the right thing. It isn’t easy, and the truth is that in most cases, the client won’t ever know that they aren’t getting anywhere near the value they should. But you SHOULD know, and not knowing, that is door #2. If you do know, and you try, and fail, then it is door #1. The choice to not fight the battles that are worth fighting, or to promote yourself over doing the right thing, that says more about you then any engagement or any situation.

So which is it?

Door #1 or Door #2?

Change the Conversation: What does “Efficiency” really mean?

One of the great mysteries of the analytics space is the use of words that have almost no real meaning. Words like optimization, analytics, marketing, social, value, personalization, predictive, and segment have different meanings to different people. They become useful jargon to direct a conversation, but when it comes down to giving them a real meaning, so many groups struggle because it is a very personal definition. When we do find a meaning for those words, it is usually an old tired one that has lost all relevance in the modern world. To me, the most commonly abused term is efficiency.

What does efficiency mean? Is it just an outcome? Is it something that you can actually measure? If it is as simple as just ROI, why do we fail then to really measure against it? I want to present a simple way to think about efficiency, your actions towards improving it, and then give you real world ways to use that to measure your actions and to improve the “efficiency” of your organization.

Here is the way I suggest measuring efficiency:

This gives you a value, which you can then measure against others. The difference between values shows you what is efficient and what is not efficient. It is strongly related to ROI, but separates its components and allows you to look at any action, not just revenue. We are given the choice to interact with 1 or all 3 parts, and we can measure our ability to do so on the same scale.

It is important to understand the 3 components, to make sure that everyone is on the same page.

Scale – The size of the population that is impacted.
Impact – This is the measure of recordable lift or gain. This is your ability to influence. This must be towards a site wide goal, not just a dependent goal such as the next page or clicks.
Cost – This is how much in time, energy, money or other resources it takes to acquire and maintain the impact listed above.

To do this however, you must always keep all three things in mind, not just one.

Scale reminds us that a high increase of a small group is often less valuable then a small increase to a large group. We can try to increase the scale of something, but without knowing the impact or the cost to achieve it, we have nothing.

Impact reminds you that you can’t only look at lift. If you hear that you got a 12% lift, then you are still missing two really important pieces of information. If the 12% is to 100 people or 100,000 people, it dramatically changes the outcome.

The cost to achieve those two pieces tells us if we actually did something valuable or not. If it takes you 2 hours and $20 to achieve this outcome, or if takes you 6 months, 500 man hours, 1.2 million in new products and has a long term maintenance cost, then it is not going to be as valuable.

In order to enforce a conversation around maximizing return, you must first change the conversation so that you are no longer discussing only one of these metrics at a time. Do not accept a conversation that only tells you lift, or that only tells you a population without knowing the ability to impact that group and the cost to achieve that change. Do not just blindly hear that you have likelihood to change a metric, understand that you have to know the cost and scale of doing so. Do not just hear that a group has a different behavior, understand that you need to know the scale of impact and the cost to change them to understand the efficiency of that action.

So this may seem like a very simple definition for a complex issue, but it gives you the ability to truly view the world differently. To quote Jim Horning, “Nothing is as simple as we hope it will be.” We like to pretend we think these things all the time and that they are obvious in every conversation, yet time and time again we drop the entire context in the name of pushing an agenda. There are hundreds of conversations every day that talk about metrics that have nothing to do with improving performance (e.g. bounce rate) or that only talk about a single portion of performance (lift). Stop those conversations, and remind people that reducing costs or increasing scale is just as effective as improving your impact. Do not assume that everyone is putting everything in the right context, because they aren’t.

So what is efficiency? It is simply the act of making sure that you are improving this ratio, and you are remembering that you can not look at only one aspect to answer a question. We can’t fail to measure actions against each other. These are not just isolated events. It is acting in a way that you keep both the denominator and numerator equivalent in your discussions and actions, and that you do everything in your power to reduce low value actions and increase high value actions. Once you have an action, measure it against other actions, and continue to balance the discovery of the value of actions against your exploitation of the higher value ones.

Being efficient is simply taking resources away from low value actions and towards high value actions. The very concept implies that you will stop doing certain actions and that you will do new ones you aren’t currently doing. It is the entire discipline of knowing that what you are doing today is wrong, and that there is always a better way to do things.

It is not the concept but the constant discipline of following it and holding yourself and others accountable that will truly define your outcome. Nothing here is revolutionary, other than eliminating all of the other factors and excuses people love to throughout in their arguments. It gives you the way to measure different outcomes against each other, and because of that, you can truly see what the value of various actions are against each other.

If you are disciplined in your tracking, honest in your impact, and willing to evaluate actions as how they help your site, and not just you, you will arrive at amazing conclusions that will shift your organization. The only way to improve is to change, so do not fear change, embrace it. Do things you aren’t sure about, challenge common thinking, do the exact opposite to see what the value of what you are doing really is. Measuring things in this simple a form is not sexy or “advanced”, and it can seem juvenile, but it is only by doing the small things well that you will ever succeed at all those large things people promise revolutionize the world.

Why we do what we do: When the Sum is Less than the Parts – Simpson’s Paradox

Some of the greatest mistakes people make is by having complete faith in numbers or in their own abilities to use them to get a desired result. While normally there are a great many just biases and logical fallacies that make up human cognition, sometimes there are factors in the real world that conspire to make it even more difficult to act in a meaningful and positive way. One of the more interesting phenomenon in the world of data is the statistical bias known as “simpson’s paradox”. Simpon’s Paradox is a great reminder that one look at data creates a fallacy that can often lead to a very wrong conclusion. Even worse it can allow for claims of success for actions that are negative in the context of the real world.

Simpson’s paradox is a pretty straight forward bias, it is when you have a correlation present in two different groups individually, but when combined they produce the exact opposite effect.

Here is a real world example:

We have run a analysis and show that a variation on the site produces a distinct winner for both our organic and our paid traffic:

But when we combine the 2, we have the exact inverse pattern play out. Version A won by a large margin for both Organic and Paid traffic, but combined it dramatically under performs B:

This seems so counter intuitive, but it plays out in many places in real world situations. You also may find the inverse pattern, one where you see no difference in distinct groups, but combined you see a meaningful difference.

In both cases, logically we would want to presume that A was better than B, but it was not until we add the larger context that we understand the true value.

While this is a trick of numbers, it presents itself far more than you might expect, especially as groups dive into segmentation and personalization. The more people leap directly into personalization with vigor, the more they are leaving themselves open to biases like Simpson’s Paradox. We get so excited when they are able to create a targeted message, and so desperate to show its value and to prove their “metal” that they don’t take the time to evaluate things on the holistic scale. Even worse, they don’t even compare it with other segments or account for the cost to maintain a system. They are so excited by their ability to present “relevant” content to a group that they think needs it, that they fail to measure if it adds value or if it is the best option. Even worse, they then go around telling the world about their great finding, only to be causing massive harm to the site as a whole.

One of the key rules to understand is that as you keep diving down to find something “useful” either from analytics or from causal feedback after the fact, the more likely this plays out. You can use numbers to come to any conclusion with creative enough “discovery”. If you keep diving, if you keep parsing, you are exponentially increasing the chances that you will arrive at a false or misleading conclusion. Deciding how you are going to use data after the fact is always going to lead to biased results. It is easy to prove a point whenever you forget the context of the information or you lose the discipline of trying to use it to find the best answer.

So how do you combat this? The fundamental way is to make sure that you are taking everything to the highest common denominator. Here is a really easy process if you are not sure how to proceed:

1) Decide what and how you are going to use your data BEFORE you act.

2) Test out the content – Serve it randomly to all groups, even if you design the content specifically for one group, test to everyone. If you are right, the data will tell you.

3) Measure the impact of every type of interaction to the same common denominator. Convert everything to the same fiscal scale, and use that to evaluate alternatives against each other. Converting to the same scale allows you to insure that you know the actual value of the change, not just the impact to specific segments.

4) Further modify your scale to account for the maintenance cost to serve to that group. If it takes you a whole new data system, 2 apis, cookie interaction and IT support to target to that group, then you have to get massively higher return then a group you can do in a few seconds.

What you will discover as you go down this path is that you are often wrong, in some cases dramatically so, about the value of targeting to a preconceived group. You will discover not only that many of the groups you think are valuable are not, but also many groups that you would not normal consider for value to be higher valuable (especially in terms of efficiency). If you do this with discipline and over time, you will also learn complete new ways to optimize your site, be it the types of changes, the groups that are actually exploitable, the cost of infrastructure, and the best ways to move forward with real unbiased data.

As always, it is the moments where you prove yourself wrong that you will get dramatic results. Just trying to prove yourself right does nothing but give you the right to make yourself look good.

I always differentiate a dynamic user experience from a “targeted experience”. In the first case, you are following a process, feeding a system, not dictating the outcome, and then measuring the possible outcomes and choosing the most efficient option. In the second, you are deciding that something is good based on conjecture, biases, and internal politics, serving to that group, and then justifying that action. Simpson’s paradox is just one of many ways that you can go wrong, so I challenge you to evaluate what you are doing? Is it valuable or are you just claiming it is? Are you looking at the whole picture, or only the parts that support what you are doing? Are you really improving things, or just talking about how great you are at improving things?

Rant – More Tests Do Not Equal More Value

I continue to see the propagation of one of the worst myths in our industry. The way to get more value from your testing program is to run more tests. That is just lazy speak for spend more money with us or give me more money if you want real results. That is a sign that you haven’t spent the time to learn how to test, and are just propagating the myths of the industry and using it to make yourself look good.

The key is to be efficient and to design your tests in a way to insure that you CAN NOT HAVE A FAILURE. I am dead serious that I can say that I have not run a test in 2+ years now that has failed to deliver both insight and meaningful actionable results that make money. In almost all cases, it took doing things that took LESS resources then what the people I was working with wanted to do. It is not magic, it is hard work, constantly finding better ways, challenging assumptions and doing everything I can every day to prove myself wrong and to find a better way. This is not about me, I have shown many people that care how to do the same, and some of them have gone on to make what I was doing at the time look foolish. I love those moments and appreciate every time they happen.

Its only when you are lazy and doing “better” testing that it is even possible to have a failed test. If you are not seeing value, change how you do things, don’t just keep throwing more resources at it until you find a story to tell your boss. So many people are afraid to challenge the thoughts of others, or to find out how wrong they are, and because of that, they just use whatever tool is fashionable to keep doing what they were already doing. If that is what you are doing, then it doesn’t matter if you do get positive lift, you are still losing almost all your money. It is not about the number of actions, but the value of the action. Doing more low value things is just throwing away money. The entire point is to figure out efficiency and to put resources in the most efficient way possible. Actions are not random, and just blaming others and resources is not the answer.

Fix what you are doing. Learn, get better, learn what about the system and stop trying to force it into some preconceived notion. You may get more if you have more time/resources, but that is not an excuse for not using what you have in the most efficient way possible. Stop making excuses, stop looking for outside reasons for your failures.

Your success and failure comes from the actions you choose or don’t choose to do. It isn’t magic, it isn’t a great conspiracy from others against you. It is your actions that get results. Stop trying to do more with more, do more with less or the same and you will find much better resources.