Personalization is such a buzzword right now that it is nearly impossible to have a conversation in the digital marketing space without it coming up. Everyone is on this quest for a “personalized experience” or to make sure that they are doing what every other group is doing. You constantly hear about all this new technology and all these new ways to accomplish this task. There is more tools and information about our users now than ever before, and yet there are very few groups or people who actually can differentiate between success and failure for personalization.

The most fundamental thing people forget about “personalization” is that I can “personalize” an experience in almost infinite ways. I can change copy, I can change work flow, I can change layout or features of the experience. Even better, I can do this for the same user in a thousand different ways. I am a returning user to your site… but I am also a user in the afternoon, who came from Google, who has been on the site 12 times, who has made 3 purchases, and who is using FireFox. So the question is not CAN I personalize an experience, at this point there are a thousand different tools and ways to do so. So the simple act of creating an experience is not the goal, the goal is to do so in the way that generates the greatest ROI for my organization.

The question needs to be, how do I discover the most valuable way to change the experience?

What we need to incorporate in any concept of personalization is a way to measure these different concepts against each other. We have to build into every process a period of discovery, using tools that allow us to know the two most valuable pieces of information when it comes to personalization:

What is my ability to change their behavior?

What is the cost to do so?

There is no way to acquire that information without actively making changes and seeing the outcome. Measuring that different groups have different behavior is easy, but what does that tell you about your ability to change that behavior? Just because one group of users purchase twice as often as another, how do you know your ability to change that behavior? How do you know that a different experience will do anything more than a static similar experience for both?

And that is the difference between success and failure when it comes to personalization. Are you just serving up an experience because you can? Or have you done the active acquisition of knowledge that shows not only that it improves performance, but that it is the best way to increase performance.

I want to give a functional example so that you can see this in action. Let’s take the exact same concept and see it executed under both ways of thinking.

Let us say that it is coming up on the holiday season, and you want to serve up a holiday shipping message to people who have purchased on your site before.

If my goal is increased revenue, then the steps would be as follows:

Discovery
1. Create multiple executions of the message (how do you know if the concept or the execution is the issue with one offer?)

2. Take 2 to 3 other messages that could be used there (one will most likely be your default content), other concepts such as specific products or specific site offerings. Hopefully you are just reusing existing content.

3. Serve all the offers to EVERYONE

4. Look at the results by segment and calculate the total gain by giving a differentiated experience:

i. If you are correct, then the highest performing recipe for the previous shopper segment will be one or both of the shipping messages. Default content would then be the winner for the non-purchaser segment (the comparable segment).

ii. If you are wrong, then any other segment will have a higher winner for any of the offers. Be open to permutation winning that you never thought of. Being wrong is always going to provide the greatest return

Exploitation
5. Push live the highest revenue producing opportunity found

Let us see how groups that get little, no, or negative value from “personalization” do the same task:

1. Push the single piece of creative to the repeat purchaser segment.

2. Hope

See the fundamental problem is that in the second scenario you have no way of knowing if it is valuable, or not. Blind belief that you are providing value is not the same as providing value. Most groups think that if they just report the outcome, or the rate of action of that group, that it somehow represents the value of that action. It doesn’t. Value only comes from the improvement of performance by that action. If you aren’t actively acquiring that information, then you have no way of knowing the value of any action. Even worse, we are adding cost and we suffering from opportunity cost from the gain we should be getting.

I want to show some simply math to show you the difference in the two groups. Let us say in the first test we have the 5 different experiences and that we are looking only at 10 different comparable segment groups (segments only matter if there is a different outcome for the comparable group). This might include things like new/returning, work hours/non-work hours, search/non search, Firefox/chrome/Internet Explorer, or any other of the infinite ways of dividing your users using any and all of the information that is available to you. You can always do more, but for the sake of argument and of efficiency, 10 different pools of the same population is enough. Segments are only valuable for targeting if we serve things to the comparable segment. If I assume that everything is purely random, then I have a 1 in 5 chance of my offer being the best. I also have a 1 in 10 chance of my segment being the MOST valuable.

(1/5) * (1/10) = 2%

So if everything is random, then I have a 2% chance that I picked the best outcome (the one that drives the highest revenue for my site), which means that in 98% of the scenarios, I have cost my site money. But let’s assume that you are REALLY good at picking segments and content based on your experience and your analysis. Having worked with nearly 300 different organizations, experience shows that the best of people who aren’t relying on causal data are no better than 2 times random guesses for choosing a better option (they guess a right answer twice as often than just the random sample).

Most groups do not fall into that category. In reality, most groups actually are worse than random at choosing the best option.

That means the math is only:

(2/5) * (2/10) = 8%

Let’s say you are the best person in the world at what you do, with great analysis and all sorts of tools, so that you are three times better:

(3/5) * (3/10) = 18%

So if you are absolutely amazing at what you do, then 18% of the time, you will have guessed the right message for the right group. 82% of the time, another outcome is better and most likely significantly better. You can reduce that to 0% of the time a better performing option with a few simple steps and accepting that we do not always understand the patterns before us. If we go back to random chance, then 20% of the time just doing nothing (your default offer) actually performs better for everyone. If you are the betting type, which would you take? 8% versus 100%? Especially when the scale of impact can be massive.

Remember that in all scenarios you are going to get an outcome, so that can’t be the measure of success. The process of finding the right answer is far more important than a conversation around the function of a tool. Nor can discussing only the impact of one segment, since we are not comparing it to others in context. The question is did doing this one thing provide MORE value than doing another action (or doing nothing), and the only way to answer that is to compare outcomes. All of the downside is when you look at “personalization” as just a function that you make a decision on and just do. All of the upside is when you discover value and then exploit it. There is nothing more valuable then when you are wrong, but the only way to discover that is when you create a system that enables it.

The difference between a success and a failure with personalization comes down to this:

If the goal is to make money, the question to ask is not to ask CAN I do personalization, but how do I put steps in place to ensure that I am having both a discover and an exploitation phase to my actions?