Why we do what we do: Why do we Shoot the Messenger? – Backfire Effect

At some time, everyone that works in data has had to deal with the following scenario:

You run a test or you do an analysis that shows that a member of management has been claiming something or pushing something that is clearly wrong. You present the data, and then they push back even harder saying that you just don’t understand or there must be more to the story. You dive back in, find more and more supporting data, you make charts and breakdowns and present them again. This time instead of just pushing back your recipient start attacking you and everything you do. They may do it overtly or behind the scenes, but they now view you as a problem and a threat. They never change their view of your original point, and now they distrust you and are looking for opportunities to attack your work.

This is a way too common outcome in the business world, and one that is not actually limited to the use of data. What you are experiencing is the Backfire Effect, or the fact that people become stronger in their beliefs when presented with evidence that directly contradicts them.

So why does this happen? Why is the data you are clearly presenting, data that multiple others agree with and buy into not having its desired effect? It is because you have started to attack their world view. Every person you ever work with believes that they do superior work, believes that they make a large impact to the business, and that they hold a deep understanding and correct view of how things work. When you present direct evidence against this, you are not actually attacking the statement, but their self-perception, which creates a level of cognitive dissonance, resulting in an ad hominem attack on the messenger, and a blind ignorance of the evidence.

Like most psychological biases the key is to set the stage for success prior to action, not after. You may not be able to force rationality into individuals or organizations, but you can certainly push discipline. Define rules of action before you start and task, work to get agreement on what will define success, and what follow up action should and will be. Often times these conversations are pushed, ignored, or dismissed, but it is up to you as the one who will ultimately be sharing the news to force this as a priority of a conversation.

No one you work with will want to talk about how you make a decision; they will want to talk about their great idea for a test, or for a group to target to, or their amazing advertising campaign. They have already decided what they want, why it is great, and what you will present in the end. If you only allow or enter the conversation at this point, you role in their subconscious mind is simply to validate their opinion. The job of those that work in data is to never give into this path, no matter how easy it is or how it may help us politically. You can not view success ever as how many actions you fulfill, but instead the value of the ones that you fulfill. The instant you allow for quantity of action to take precedence over quality of outcomes, you are setting yourself and others up for this type of failure. It is instead to be the holders of discipline, to be the ones that help create opportunities to find out the faults in these ideas, to not be the ones to validate held world views.

This is also why changing the conversation about what it means to be “right” and “wrong” is so important. If you shape each conversation to talk about the amazing outcomes of being “wrong”, of going in a not previously encouraged direction and about the impact to the business, you are opening the door for individuals to not have their world view attacked. If you allow others to understand that they have impacted the business, and that they have succeeded in their end goal of finding out people cases where they are wrong, you have enabled them to not fall into the Backfire Effect. Changing the conversation away from the faults of one idea and towards the value of different options and why choosing this action allows you to not attack someone’s world view and instead help them look good by giving them the tools to find an outcome, not just an input to a failed system. It is important that you understand deeply why you need to do this, what the traps are, and what the right way to frame that conversation is, but if you are willing to do the ground work you can achieve amazing results.

One of the defining characteristics of organizations who get value from their data versus those that don’t is that the leaders who manage their data focus on the leading conversation, not on the stories they can tell after their analysis. This problem is only exasperated by egos and by the fact that so much of the material and talk in the industry is filled with justifications for those that do not want to address the real issues at hand. Much of the data marketplace, from managers to agencies, is filled with those that would come up with creative ways to tell people exactly what they want to hear and to come up with a story that shows impact, even if there is no factual basis for that claim. There are articles, speakers, and “experts” throughout out who have mastered the art of sounding intelligent without actually adding anything new or functional to the organizations of which they address. There are many groups who have their own biases in believing their value is presented, just like any other group, because they focus simply on the actions some takes or on their ability to make a recommendation. your key responsibility is to focus the same skills and control the message in the same way towards that which will actually drive value for the organization, not that which sounds good but is hollow. It is vital that from day one and onwards that leaders control and help shape the conversation instead of responding continuously to requests. Successful organizations define actions and successes, focus on discipline, and prepare for action before the data, not after.

There is no more true statement then: “Success and failure is determined before you act, not after.”

There is zero chance of you avoiding push-back if you fail to do the dirty work of setting the stage properly. If you create an environment where you don’t focus on the idea but instead on the discovery, on the outcome and not the input, and work with groups to add value to their ideas instead of facilitate their ideas, you will find amazing results achieved throughout the organization. Ultimately you need to be agnostic about what wins and loses, and instead focus on how people arrive at a decision and if it answers the correct business question. Shy away from this aspect of the job and you face the challenge of dealing with the backfire effect or finding ways to justify actions you rationally know are not valuable.

If you want to avoid painful confrontations, you will always have two options. Option one is the easier one, convince yourself that presenting data that supports people politically or that just getting someone to act is somehow providing value. In this option you will never be delivering news people don’t want to hear. The second option is to focus on the painful disciplines prior to actions and to deal with some discomfort before you get too far and stay away from anyone’s ego. In this you will have to deal with some discomfort, but you will be able to make a true and meaningful impact to your organization. Other departments, executives, and even your own management will never be able to make this decision for you, this decision is a personal one and one that you either choose to make, or one that is chosen for you.

How Analysis Goes Wrong: The Week in Awful Analysis – Week #9

How Analysis goes wrong is a new weekly series focused on evaluating common forms of business analysis. All evaluation of the analysis is done with one goal in mind: Does the analysis present a solid case why spending resources in the manner recommended will generate additional revenue than any other action the company could take with the same resources. The goal here is not to knock down analytics; it is help highlight those that are unknowingly damaging the credibility of the rational use of data. What you don’t do is often more important than what you do choose to do. All names and figures have been altered where appropriate to mask the “guilt”.

I have a special place in my heart for all the awful analysis that is currently being thrown around in regards to personalization. So many different groups are using personalization as the outward advantage prevalent with big data. No matter where you go, ad servers, data providers, vendors, agencies, and even internal product teams, they are all trying to talk about or move towards personalization.

This is not to say that personalization is a bad thing, I believe that dynamic experiences can produce magnitudes higher value then status experiences and have helped many groups achieve just that. What most surprises me however is the awful math being used show the “impact” of personalization from groups who have achieved absolutely nothing. I have lost count the number of times I have walked into and found one person or group talking about how personalization has improved their performance by some fantastic figure that it seems that the business should be doing nothing but thank them for their genius. The sad reality is that most of the analysis is biased and bad that in most cases the same companies are actually losing millions by doing this “personalization” practice.

Analysis – By putting in place personalization, we were able to improve the performance of our ads by 38%.

We have to tackle the larger picture here to evaluate statements such as above. Before we dive too deep into how many things are wrong with this analysis, we need to start with a fundamental understanding of one concept. There is a difference between the changing of content or the user experience and the targeting portion of that experience. In other words, changing things will result in an outcome, good or bad, and then targeting specific parts of that change to groups is also going to lead to an outcome. The only way that “personalization” can be valuable is if that second part of the equation is the one leading to a higher outcome.

1) Just to get the obvious out of the way, the analysis doesn’t tell you what the improvement was. Was it clicks? Visits? Engagement? Conversion? Or RPV? If it is anything but RPV, then reporting any increase has no bearing on the revenue derived for the organization. Who cares if you increased engagement by 38% if total revenue is down 4%.

2) The only way that “personalization” can be generating 38% increase would be if the following was true:

The dynamic changes of content raised performance by 38% to total RPV over any of the specific static content or content served in ANY other fashion.

In other words, if I would have gotten 40% increase by showing offer B to everyone, then personalization is actually costing us 2%.

3) Since most personalization is tied to content and the inherent nature of content changes is very high initial difference and then normalizing over time, what is the range of outcome? What is the error rate? The inherent nature of any bandit problem with would use causal data to update content means that you either have to act as quickly as possible, resulting in higher chance of error, or act slow and risk the chance of not responding fast enough to the market. In either case, performance will never be consistent.

Rather than continue to dive through each and every biased and irrational part of this analysis, I want to instead present two ways that you can test out these assumptions to see the actual value of personalization:

Set-up: Let’s say that you believe that 5 different pieces of content are needed for a “personalized” experience. In other words, you have a schema that will change content by 5 different rules.

The same steps work for anything from 2 rules to 200.

Option #1 (the best option):

Serve all 5 pieces of content to 100% of users randomly and evenly. Look at the segments for the 5 rules AND all other possible segments that make sense.

You will get 1 of two outcomes:

1) Each piece of content is the highest performing one for that specific segment and those are the highest value changes


2) ANY OTHER OUTCOME which by definition in this case results in more revenue.

Option #2

Create dynamic logic in the tool, based on the 5 rules.

Create 6 experiences.

Each experience except the last shows each piece of content one at a time to all users (so content matching group A actually gets served to all 5 user definitions in recipe A). In the last recipe, then add the dynamic rules to the last experience.

If the last experience wins, then you at least know that the dynamic content is better than static content. If you are looking at your segments correctly, you will then also be able to calculate the total lift from other ways of looking at the content to the dynamic experience that you tested. If the dynamic experience is still the top performer, congratulations on being correct. If any other way works best, congratulations on finding more revenue.

In both of these tests, if something else won, then by doing what you were going to do or what you would otherwise report on IS COSTING THE COMPANY MONEY.

There are massive amounts of value possible by tackling personalization the right way. If you do rational analysis that looks for total value, then you will find that you can achieve results that blow even that 38% number out of the water. Report and look at the data the same way that most groups do though, and you are ensuring that you will get little or no value, and that you are most likely going to cost your company millions of dollars.

How Analysis Goes Wrong: The Week in Awful Analysis – Week #8

How Analysis goes wrong is a new weekly series focused on evaluating common forms of business analysis. All evaluation of the analysis is done with one goal in mind: Does the analysis present a solid case why spending resources in the manner recommended will generate additional revenue than any other action the company could take with the same resources. The goal here is not to knock down analytics, it is help highlight those that are unknowingly damaging the credibility of the rational use of data. What you don’t do is often more important then what you do choose to do. All names and figures have been altered where appropriate to mask the “guilt”.

If we were to really dive into the real world uses of data, there are two parts to every recommendation that an analyst makes. The first is that action is needed, and the second is what action to take. Fundamentally the problems arise when we confuse which one of these we have actual valid data for, and even worse when we convince others based on those flawed assumptions. While the core goal of analysis is to encourage action, we are creating a duplication of the same flaws that analysts rail against if we are presenting a course of action that is not based on factual data but instead on our own biases and opinions to which we simply attach data as a means of justification.

A perfect example of this is a very common and easy ROI analysis across channels. The multitude of problems with attribution are too long to get into here, but needless to say this is yet another example of people confusing rate and value, or more specifically attribution does not mean generation. Because of this, you can easily take this type of analysis, which can make a case to for action, as some sort of determination of what action to take.

Analysis: By looking at our different marketing channels and the revenue that we attribute to them, we find that Paid Search has an ROI of 145%, Display 76%, Email 250%, and Organic Search 112%. Based on this, we recommend that you move as much of your budget away from display and towards email and Paid Search.

The case you are making here is that you need to optimize your spend or that you can in fact make more money by improving what you do. I find this an ironic statement however in that I would hope that every group knows that they need to constantly improve, and those that don’t know that are not likely to accomplish anything if they do act. If that is not the case, then one must question what the point of any argument is. It is either an argument for the sake of avoiding either blame for past incompetence or pushing solely for the case of presenting evidence of growth, even if there is not any functional improvement. In either case, the story is secondary to the suggested actions derived from the data.

The real problems here lie completely with the suggested steps to improve. Let’s dive into the components of it:

1) Just because I can attribute 250% of revenue to email, it doesn’t mean that I actually GENERATE $2.50 for every dollar I pump into email. The problem here is that you are simply saying that people who interact with email ended up giving us X amount of revenue. You have no way of knowing if it was the email that lead to that revenue, or if people who make purchase and plan to again would be the same one’s signing up for more communication.

2) You have no clue if these channels are even causing revenue at all. It is possible that by not showing someone a paid ad for a specific item, that they would instead purchase a different item and generate more revenue. Even if you do not believe that is likely, you can’t in any way know how much of the revenue is generated solely from the channel.

3) Cross pollination of people hitting multiple channels is in here, so you had to pick an arbitrary method for assigning value. No matter what you choose, you are adding bias to the results and adding more confusion to the outcome.

4) Changes are not linear, so even if we moved revenue from one to another, you don’t get linear outcomes. You might not make a single dollar more.

5) You don’t know what is cannibalizing the other sections. It’s possible that paid is taking away from organic, or display from organic, etc…

6) Because you don’t know what is generating revenue, then it is just as possible that display is generating revenue more than any of the other channels. While I hate anything that looks like it isn’t even breaking even, if the analysis is to cut the lowest performer, we have no measure of what the lowest performer is.

7) The entire analysis doesn’t even look at correlation between spend fluctuations in the analysis, which is far from perfect, but at least can start to look at what incremental value you get from adding or reducing spend.

Marketers and managers love this type of analysis because it makes an easy story and it seems to have a clear action. The reality is that it can be the most damaging, not only to the rational use of data, but also the company because the story that is presented has no basis in reality. You could be right and nail exactly the best way to change, or you could be dead wrong, or somewhere in between. The sad reality is that if you keep the conversation solely at this level, then there is no way for you to ever know what the real outcome is or for you to be able to justify anything you do or say afterwards.

The Joys of Being a Consultant

The life of working with organizations to bring about change is a difficult and frustrating one, but at the same time there are some truly amazing moments. The times you see an organization change and truly look at optimization differently, when they get past misconceptions and “best practices” are truly amazing. While much of my writing focuses on the dark side of this world, I did want to take some time and share a few small stories from the front-line that are what to me consulting is all about.

Story #1: I was working with a hospitality and travel company where I was hoping to come in and change a number of practices that may not have been beneficial to the company. Starting out an example test series to work through discover and exploit, they decided to run an inclusion/exclusion test on a page template for their destination landing pages. We discovered that their main content block section was having a negative impact to their page and that removing it would result in 12% lift to RPV. We also discovered that a small section on their right rail that they never even thought about was actually the most influential part of the page.

We were able to do this with very little effort (basic CSS), but because we challenged a number of internal assumptions, we learned about things they never considered, reduced maintenance cost, and achieved a major lift to their bottom line. The real kicker of this testing was that it was both the lowest resource test they had ever run and their most valuable.

Story #2: Working with another travel/hospitality organization, I came in because they had not been able to get any momentum with their testing due to a previous belief that it required too much resources. We were able to leverage their existing infrastructure and ran a simple inclusion/exclusion test on one of their landing pages. We did this all in one conversation without any pretext. In this case we discovered that almost every element individually was negative to the page, meaning that as a whole the total was far less then the sum of the parts. We run a simple follow-up and discovered that they could generate 3-8 million dollars by simply removing one of their main offer parts, and that they could generate 4.5-12 million if they were willing to create a dynamic experience and show one of two sections based on new and returning users.

I love these types of moments because you have so many good things come out of very small actions. We now have a group that is amazed by the power of optimization. We now have a group that sees that a number of their assumptions have been proven incorrect about what works, or even for whom. We were able to generate massive fiscal impact to the business, with extremely low effort, and were able to give them insight into other opportunities to do the same thing. They were able to learn about the disciplines of testing, find out that conversion rate and revenue are not correlated, and were able to think about their entire user experience differently.

Story #3: Working with a major online retailer, they were just re-introducing optimization to their organization. They had already decided on what they were going to test, how, and when. They then jumped into their test, but I gave them guidance to not over react and to understand how contextual changes are going to play out (very large initial climb and then narrow down to almost nothing quickly). They were so excited when they had a 40+% lift in a couple of days with 99% confidence, but I asked them to wait and talked to them about what confidence really means. They held off and within the next 7 days, the results of the test were less then 2% to the point it had no impact.

Now normally you don’t like outcomes that do not generate lift, what made my day is that they stopped themselves from rushing around claiming a result that had no basis in reality. If they had ran to tell the rest of the organization about that 40+% lift, they would have done irrecoverable damage to the company as others would have attempted to do the same thing and reach the same false conclusions. Because they waited, they get to understand the nature of statistics, contextual versus spatial changes, and started thinking more about the true disciplines of optimization. Since that time they have now expanded their testing to multiple countries and are testing at an accelerated pace and driving real value throughout the organization.

There are so many moments in working with different groups where you can get frustrated, or give up, or even worse give in and tell someone what they want to hear. Equally there are many moments where you can create a false impression of impact and fake numbers for your own gain. The moments that are truly amazing are those that prove that you don’t have to do any of those things, that you can achieve amazing results, quickly, and for far less time and effort then someone might otherwise think. All it takes is to think about and tackle problems wrong, and the results that any group can achieve are truly amazing.