One of the most difficult principles to understand for many people in our industry is that rate and value is not the same thing. One of the fastest ways for a program to go astray is to confuse one for the other. It is easy for people to understand the need for agreeing on what you are trying to accomplish, or why they need to have leadership. It is even easy to talk about the need for efficiency and that it is ok to be wrong, but yet even when people get past that point, they still consistently miss this critical difference. We so desperately want to explain our value to the company, that we confuse the value of our actions in an attempt justify our actions. This fundamental loss of understand leads to a wide range of poor decisions and bad understand that dramatically limits the positive impact a testing or analytics program can have.

A rate is simply a ratio or a description of actual outcome; it is the same thing as me telling you that I have a $4.23 RPV for a population or that I got 5000 conversions. This is a description of past behavior, and is simply an outcome, not a description of why or how that outcome came to be. Where people lose focus is that the value, or ability to positively or negatively influence that outcome, is not tied to those gross numbers. A description of rate tells you nothing about an individual action, since you are not comparing that outcome, only describing it. Increasing your conversions does not inherently create more revenue, nor does the revenue by itself reflect positive value generated by an action. We measure things by saying we ran a campaign and then we got $3.56, this is not the same as telling you anything about the value of that campaign. Value would be the difference in running that particular campaign versus not doing anything, or running a different campaign. The rate is the end outcome, the value of that action is how much it improved or decreased performance.

People are so conditioned to express their contribution or to explain their value as the outcome of a group. I am responsible for the product page, or SEO, or internal campaigns so therefore I must be the sole reason for the generation of that value. Just because your department or your product produces 10 million dollars, it does not mean that is representative of your value. Value is simply what would have happened if you had done nothing, or if you chose a different route. Value is what we are really talking about in optimization, we are discovering the various tools and options that allow us to influence our current state and improve performance. We have a way to measure the efficiency of different actions and choose actions based off of a rational process instead of opinion and “experience”. The value of an action is the amount it increases or decreases the bottom line, which means your value is the ability to choose the best influencers, and avoid the worst ones. Stop defining actions by the rate and instead think in terms of value and you will completely change your view of the world. The question is never what did it do, but would it have done if I simply stopped to exist, or if we chose any of the other routes available to us. Which one would have generated the highest possible outcome?

So where does this cause people to go astray? The first place is in assuming that past behavior reflects value, instead of the rate of action. People are so used to doing complicated analysis that shows that people who click on section Y are worth $3.45. What they are missing is that you are expressing the rate of revenue from people that click on that section, not the value of the section. Using correlative information it is impossible to know what that is really influencing, is it positive or negative? Would you make more or less from having it? Or what about other alternatives? It is a fundamental shift in how you view the world, not focusing on what was, but only focusing on the influence and cost of changes. Getting caught on this definition often leads to misallocation of resources and groups holding items sacred that are negative to end performance.

This change in viewing the world also requires also that every person accepts that their inputs, skills, and responsibilities are part of this system, and that it is rarely going to be a perfect match between what is best for their group and what is best for the organization as a whole. You are not defined by the rate output of your responsibility, but what you do with it. What matters is the ability to view everything as working together to improve the whole, which necessitates the need to not focus on individual groups, items, or interactions. When we are trying to generate value for the organization, and improve our bottom line, the least important item is what does that do to item X, or to section Y. That information is rarely meaningful as it is a single data point, is always going to cause a cognitive dissociation with what is best for the site and the whole. Getting people to act rationally is not inherent to the human condition, but is vital to getting the best results.

The easiest way to prove this simple dissociation between rate and value with testing is to do an inclusion/exclusion test and simply remove each item one at a time. If you know the rates before hand, or if you believe that an item is worth some value, then it would mean that you would drop that entire value when you remove the item. In reality, you will find little connection between that correlative value and outcomes, and will be shocked by how often you find things that you thought were valuable, but that turn out to be negative to the total page performance.

Testing is an amazing tool that allows you the ability to see the value of items. It is not very useful for the rate of outcomes, since we are trying to compare outcomes, but it gives you so much more insight then what you had before. It frees you up to see the world different and to tackle real problems that you could never tackle before. Understanding what information is telling you, what it isn’t, and how best to leverage different types of information together is what changes myth to reality in your use of data. In order to start down that path, you must first deeply understand the difference between rate and value, and understand that your job is not to focus on rate, but instead to discover value.