Why we do what we do: How to not live for today – Hyperbolic Discounting

I was having a conversation with a colleague about the best way to assist a customer with their 2012 planning, and it brought up the third bias I deal with on a common basis, Hyperbolic Discounting or “the tendency for people to have a stronger preference for more immediate payoffs relative to later payoffs, where the tendency increases the closer to the present both payoffs are.” Besides being the basis for the entire credit card industry, in optimization this mostly comes out in terms of better versus best testing.

When groups decide on a roadmap, or the list of tests that they mistakenly refer to as a roadmap, they will often prioritize them by what is most on top of people’s minds, or what their boss wants. You have all been sitting around and think that changing the button to Buy instead of Buy Now, or you really want to change the background on your promotional images on your front door, or the copy on your landing page, you think it needs to say X… All of this leads to the want to test your theory and prove yourself right. You shut down, ignore discipline, and try to see if that idea is better than the other. The entire concept of hypothesis testing leads to this failure of human cognition. It works if the only goal is to prove a single point, but it is really inefficient if the goal is measure the relative value of an action. We forget that we are dealing with more then the day to day issues in front of us, and we try to solve a problem today instead of pushing to get the most from our entire program. Even worse, it leads to the prioritization of these tests over more efficient and discipline based tests because of the immediate payoff in the “I am right” reward they offer.

The reason that this fails is because the focus is on that short term gain, we often think it will take too much effort, both resource and especially politically to try to learn in our efforts. Instead, you end up down the path of simply trying to figure out if version B is better than version A. It isn’t about figuring out what you should be doing, it is about trying to prove your idea better then the other persons idea. In some cases, groups will add a few smaller tweaked versions of A or B (which is still better), but end of the day, your test ends up answering, “Which is better, version A or version B”. If the concept is bad, it is an inefficient test. If the section you are optimization is not as important, version B can be massively better then version A, but still only as valuable as the least “better” version of a different section of the same page (or a different type of change to the same item), it is still an inefficient test. It is far more valuable to know that if I spend $5, I can get $15, $7, $20, or $30 then to just know I got $7 and be happy with it. True optimization is to figure out the best path, not to just measure the one that you are already on. Yes, you get a result, but it is a really inefficient result, and often leads to further inefficient results as you continue down that path. Even worse is when you try to force a MVT to just throw multiple better test together and simply increase the speed of your sub optimal outcome. There is no way to make this type of testing efficient, unless you get ahead of the problem and can deconstruct the question of the test.

Best testing takes a different approach, it asks much more fundamental questions, such as “Does my copy matter the most on the page?”, or “which factor of the button is most influential?”, “what are the feasible options”, or even “does changing the button, is that the best place to put my resources?”. Best testing is about figuring out where the best places are to focus your energy and the best feasible alternative; democratizing the entire process so that your outcomes are less biased by any individual or idea. Best testing becomes about the system you have in place to make decisions, and not about any individual idea that goes into it. Any system is only as good as the quality that goes into it, but it designed to maximize your returns that come out of that system. It also makes you deconstruct your questions, one of the single most important skills for a successful program, and challenge core assumptions and biases that you have. The key is to figure out what places and what types of changes are BEST relative to each other, so that you can align resources and internal thought towards those points, while eliminating waste on things that do not matter. The act of valuing possible actions against each other with causal information is the single greatest way to maximize resources, not use them unnecessarily. It challenges you to look at the page and user experience as a holistic item, and that each component is part of that process, and then to weigh the value of each one AGAINST each other. It isn’t about your idea, or even any individual idea any more, it is about creating a system by which you can learn and figure out where to focus your energy to get even better concepts and even better outcomes. Who cares if the copy improves the page if it only does it at 1/10th the scale of changing the background, or the main image, or that small section on the page that you never expected?

What is amazing is that you are left with two options from the BEST testing:

1) You were correct – In which case you have confirmation and you still get the same lift.

2) You are wrong – You learn valuable information, and you get MORE lift then what you had before. Even better, it might send you down a path that you weren’t even considering before.

What happens when you are stuck doing better testing, is that you get the short term return, both on your idea, but also politically by showing that you were correct. You are ignoring the long term opportunity cost of figuring out what matters most and building off of something. It gives you a shiny object that you get to show off to anyone that will listen, but the real question is what are you giving up going down that path?

The irony of this is that everyone always thinks that that will take too much time or resources, when for a large majority of sites out there, it is almost always the exact opposite. Being efficient means that you are testing based on your current resources and the way to maximize speed, where better testing often leads to forcing an overly technical solution in order to execute the one idea that you are willing to consider. What is true however is that you have to shift how people think, and challenge them to understand some new disciplines, in order for them to accept and execute in this way. It takes a culture and a person willing to be “wrong” and one who is willing to let go of their ego for the sake of the site. To paraphrase Kathryn Schulz, “you are already wrong, the difference is simply that you will know that you are wrong”.

What you get out of your optimization program is all about the system you put in place and how much you are willing to challenge yourself and others to find the best thing you can be doing. It is easy to want to take the immediate payout of being right or proving a HiPPO right, but at the end of the day, you will always be better off if you focus on the disciplines that make you successful. The best thing that testing can do is challenge or eliminate human biases from the plate. It can create an equal playing field so that you can correctly know the causal value of an item, and to be able to measure the EFFICIENCY of improving it. It allows all ideas to be measured against themselves, and more importantly, it stops groups from spending hours talking about ways to improve things that don’t matter, while teaching what does matter and how best to impact it. It doesn’t eliminate the value of good input; it just filters it and refines it so that it is spent on the correct things, not just what person A thinks is the most important thing.

Minority Report – How to avoid failure for personalization

One of the largest pushes in our industry over the last year is to create a massive personalization scheme on your website. Apparently people missed the point of Minority Report, because the movie mocked this behavior and people’s assumptions about what people are going to do. Time and time again, I hear about clients who are sold on “personalization” who have built out massive 35 and 48 point schemas about where and what they are going to target their personalized content, only to have them be shocked when I talk about how inefficient that is. Without fail, the clients who get the worst return on their optimization efforts are the ones that push full steam down this boondoggle without applying discipline or feedback into their efforts. Dynamic experiences built to meet user needs are an amazingly effective tool, but in order to get that value, you must first tackle your own assumptions.

You are making three massive assumptions when you just run down this road:

1) Assuming you know WHO matters most
2) Assuming you know WHAT type of changes matter to them
3) Assuming you know WHERE to make those changes

I would add a 4th one:

4) Assuming that the full cost of personalizing your website is much less then it really will be.

Who are you targeting to?

Often groups want to push out based on a site behavior or behaviors to try and create a more “dynamic personalized experience based on what the customer has already declared their intent is”. Things like changing the main banner based on the number of visits or an engagement scoring system. Groups will sit around in giant meetings trying to come up with the perfect scheme, which is counter productive to getting the very value that those meetings are hoping to achieve. Just deciding on what you are going to do completely misses the point in that the same person can be looked at 100 different ways if you so choose. That same person came from somewhere, looked for something, used some sort of browser, during some day of the week, at some time, with some sort of prior history… You can choose to target to ANY of those things, which means that you need to figure out a way to measure the value of each one AGAINST each other.

This gets us to the point of what I usually refer to as an “exploitable segment”, one that has shown through CAUSAL data that it creates a change in user outcome based on a change in the user experience. This is one of the many differences in the disciplines of analytics and optimization, since in the world of correlative data; you are looking for groups that have a different behavior. In optimization, we are looking for groups who CHANGE their behavior based on a test. This means that we don’t care that people who come from search spend half as much time on your site as people who come straight to the site, we only care if those two groups have a different “winner” for various test results. If the same thing wins for both, they may both have a different propensity of action, but you gain nothing from changing the user experience relative to each other; the same thing helps both.

In the worst case, groups start out by talking together or letting one person come up with a concept or schema that they are just sure will work. In the next step up we often we deal with groups strictly rely on CORRELATIVE data to try and answer this problem. Unfortunately, correlative data can’t tell you the value of an action nor the efficiency of an action. The best it can do is give you some insight into the rate of action and the likelihood of an outcome as long as you do not factor in cost or efficiency. Analytics data can tell you what the rate of action is for the search people, but it cannot answer if their behavior will change based on a dynamic experience. You need to use causal data, the value of the changes relative to each other, to really dive in and discover what groups are actually going to change their behavior and that you can leverage to improve site performance.

What Matters?

I cannot stress this enough… a user experience is more than banner content or copy. Usually the largest and the longest lasting changes to sites are based on real estate, or changing the spatial dynamics of a page either through layout, inclusion/exclusion of items, relative positioning of items. Your other options are changes to presentation (how something looks) or function (how it interacts or how it is programed) and then copy/content. All of these things can be changed, and figuring out the order of value is vital to efficiency. One of the very first things any program needs to do is figure out the value of different types of changes relative to each other. You have to think in terms of the entire user experience and then figure out what changes, relative to each other, are most valuable. Even if we are limited to one content area, we can look for efficiencies of scale by focusing on the components and rules of the content over the individual content item, in order to gain returns on otherwise perishable changes.

Where should I personalize?

Most teams walk into personalization with the want to tackle the entire site in one blast, to have the same type of personalization on the front door, landing pages, product pages, in cart, in the right gutter, everywhere where they can fit it in. In most cases, even if one of those or multiple of those spots is positive, the overabundance is inefficient to the point of counteracting a lot of the good that may come from putting personalized experiences up on the site. It also fails to account for the fact that maybe those spots shouldn’t exist? Or maybe different functions on the site need different types of “personalization”? Or maybe the most personalized thing you can do is move items on the page to increase the efficiency of the user flow. Like everything else, it requires study and discipline to figure these things out.

So often the first thing I do when working with groups is ask them to prove out the value of what they are targeting; without fail what they are doing has no positive value, and can often times be hurting performance. You have to be disciplined to get real, long term value, from your “personalization” program. Optimization gives you the ability to measure efficiency and the value of items relative to each other, which means the same tool you want to target with can give you so much more, with almost no effort, simply by being willing to ask this fundamental question. We are trying to learn about what matters most, not try to be “right”. Nothing is more valuable then when you are “wrong” about your assumptions.

Like so many things in the past and that will be, personalization is the next buzzword bingo item that has caught the attention of the online world. It doesn’t mean that the concept is not valuable or is not something you should strive for. Some of the sites that get the most value get it from dynamic targeted user experiences. What it does mean however is that you cannot jump in without understanding the discipline it takes to achieve real long term success. Without being willing to go down the path of learning, almost all efforts are doomed to failure.

I fully encourage you to find meaningful, exploitable, dynamic user experiences. You need to work to make your site a living breathing thing that shifts to meet new needs and is something that different people get different things from. What has to happen though, is you need to tackle each step with the attention that it requires and to apply discipline to reach meaningful results. You cannot just guess your way to victory, but you can get there easily if you are willing to answer key questions through action and allow the results to dictate the path you are on, not your own ego.

Why we do what we do: Living in the fishbowl – Expectation Bias

In my quest to tackle the major biases that plague our industry, the second one I want to tackle is Expectation Bias or “the tendency for experiments to believe, certify, and publish data that agree with their expectations for the outcome of an experiment, and to disbelieve, discard, or downgrade the corresponding weightings for the data that appear to conflict with those expectations“.

In statistics and math, you deal with the concept of expected outcome. You take the payout, factor in the percent chance of it happening, and then factor in the number of chances you are taking to reach there. This is the exact reason that the lottery is such a bad investment, because it has a massive outcome, but such a low likelihood of reaching that conclusion that your expected payout is never going to happen (Neglect of probability bias). Just because one person did win the lottery does not mean that most people win the lottery. The same is true for just reading the stories or copying others. There is a want to push out and tell people about how advanced you are, as well as the want to see immediate payoffs (hyperbolic discounting). There is also the need to distinguish yourself or your offering and to discount others if an action only works 1 out of 20 times, and even then you don’t know if it is the best outcome even for that group, what possible good is it going to do you?

Obviously, the problem then becomes knowing the likelihood and the relative value of the outcome…

Good luck getting that information if you don’t have direct interaction. Not only are people hesitant to share if they know, but people are inherently wired to not seek or know that information themselves (choice supportive bias). What does happen however is a giant group of ankle biters… people or groups (especially agencies or “experts”) who promise that they have all the answers, who tell you that if you listen to them, everything will be golden, and that all of their clients are super successful. It is not evil, it is not even that they know what they are acting selfishly, they are just wired to think in these terms, as any human is to want to believe them. This phenomenon and the play of it on the human psyche is a recurring theme and one that leads to very inefficient and under performing programs, where people are left either defending poor results or seeking yet another magic answer to all their problems.

Successful consulting and successful programs focus on the disciplines of success, not just the actions. They talk about both good and bad outcomes, and how to get more of one and less of the other. They focuses on what defines successful actions and alternatives, not just what you are doing or a single possible alternative. Challenge anyone or anything that only talks about “success stories” or who can’t talk about how often things work and how that compares to alternative processes. There exists a feedback loop where bad practices are shared and championed just because they are being acted on, and not because they are valuable. It allows them to tell stories that show that they got someone to do… something. It is up to you to stop your path down that mobius strip, and instead challenge yourself and others to think differently and to think in terms of finding a better way and a better answer. Programs have to be able to learn, grow, adapt and deal with real world problems, which means that any single “answer” is just playing to your hubris and leaving you open to future failure.

One of my favorite sayings is “Information is not knowledge”. Seeking out better information and challenging the stories and practices of others, challenging your self to learn and grow and change your biases and your want to be “right” is the only way for your program to truly move forward. Focus on the system, focus on learning, focus on fixing why you do what you do, and never stop, never let go, and never let yourself fall for these traps. That is how you succeed.

Why we do what we do: Garbage in and Garbage Out – Congruence Bias

One of my favorite online blogs is You are Not So Smart. I find it fascinating the fallacies that make up our day to day lives, and find that the number one driver of changing client cultures is to tackle and teach lessons that help them challenge or break some of these fallacies.

The first of these biases is CONGRUENCE BIAS, or “the tendency to test hypotheses exclusively through direct testing, in contrast to tests of possible alternative hypotheses“, or otherwise known as, trying to prove myself right. You see this all the time in testing groups, as they test only 1 or 2 things that are very similar, then get a small win and claim it is the greatest thing since sliced bread. You also see this when they know they are going to test targeting to return users, or changing the product image on their landing pages, or any predetermined priority of elements or changes. We are all guilty of this, and it takes discipline and dedication to not let it run your program or your life.

There are entire sub industries who feed and live off of this fallacy, like sites that show winning tests, many “top” blogs, or people or agencies who sell “we have all the answers”. Even internally, how often have you sat around having “I feel/I believe” conversations to determine what the best course of action is (and as a side, why do people think those two statements mean something different in that context?). You also see the same thing when you have groups who all they want to do is talk about tests, not about their program. It means that they are so caught up on trying to prove themselves right that they are missing the forest for the leaves.

It is fascinating how often the groups that get some of the worst ROI for their testing are the ones who champion and talk about it the most, and the reason for it is this bias. The people in charge are trying to make themselves look good (I had the right answer) instead of finding the best answer. The answer is simple… know you are wrong… even if you have the perfect “best” answer today, you are going to be wrong in a number of months. If you don’t have the context for the result you have, or if you don’t hear the context for the result that someone is pushing on you, then you do not know anything about the value of the outcome.

If you want to make sure that you are both getting the most value, and that you are letting a program learn and grow to the best conclusion, the choice is simple, challenge this bias. Include the null assumption, include every feasible alternative, even if you think it won’t win (there is an almost inverse correlation between what people think will win and what will win). Challenge that just because one person out there got a 10% lift doing something, that it is the best thing for you? How do you know how many other people failed miserably doing it? How do you know that even if they got the 10% that they couldn’t have gotten 15 or 20% by doing something simpler and with less effort?

Challenge yourself… Challenge how you think and challenge how others think and you will find the next “truth”. Let the reality of the data (the CASUAL data) tell you where to go and what to do, don’t let hubris or popular opinion tell you the value of something.

The system is what gets you values anyways, not any individual an action.