Why we do what we do: You are Not so Special — The Forer Effect

While the quest for personalization may be newer to some members of the online marketing world, the reality is that is a concept that is old as sales. People have been trying to convince others that they alone were getting a special deal or that their message was meant just for them. One of the great practitioners of this concept was P.T. Barnum, who famously billed his circus as “we’ve got something for everyone.” On some level everyone understands the appeal of being special and of having someone take the time to tell me something that is unique to just me. The greatest salespeople though understood one of the great ironies of personalization, which is that general statements, when given in context, often are treated as deeply personal and are extremely powerful. This concept is known as the Forer effect, or more directly, the tendency of people to interpret statements as being accurate for them personally, even when they are not.

The Forer effect gets its name from B.R. Forer and came about from a series of experiments that he performed in 1948. His famous study involved giving a personality test to all of his students. He told them that they were all receiving unique personality analysis, and they were to rate that analysis on a scale of 0 to 5. All of the students actually received the exact same results, using such lines as, “While you have some personality weaknesses you are generally able to compensate for them.” and “You also pride yourself as an independent thinker; and do not accept others’ statements without satisfactory proof.“. Despite the exact same statements being made, his students average score for his “analysis” was 4.26.

The favorite trick of psychics, conference speakers, and astrologists, this psychological bias is important to understand, especially when thinking about the concept of personalization. General statements hidden to look like targeted messages have much greater impact than direct statements, and are far more likely to increase belief in the speaker. Personalization, as it turns out, is about not being personal, or at least not to the Nth degree. Personalization is about the match of the general and the pseudo specific, and it is about taking that message to the largest group possible, not just the ones that directly match the message. The more we can measure different types of messages, and the more we can find the largest groups that respond to them, the better our results, since functionally the more targeted the message, the less overall gain we get to improving total site performance.

So you might ask why is this so important in the quest for personalization? This bias tells us that overly thinking personalization and designing a large number of specific messages is both a waste of resources, but also far less likely to create a positive outcome. It also tells us that the message itself does not have to match the rules that dictate the outcome; general statements have an impact for a large variety of people, not just a specific targeted group.

As you start thinking about and tackling your personalization programs, it is important to understand the nature of why you are doing these actions. Barnum knew that he was there to sell his circus, and every action he did had only that outcome in mind. He was one of the most famous practitioners of a single success definition, and he knew that no matter what he did needed to drive more to spend more on his circus. The same is true of all online efforts. Your goal in the end is to make more money, and the key is not to focus on a specific message, or to over rely on experts or correlative information to tell you when and how to target. The key is to test out all sorts of possible content, and to see how you can best present them to people to allow you the efficiency of largest group of people possible.

This is why a message about a specific product may work best for Firefox users, why time of day may be the best match for your re-targeting content, and why smaller segments are so inefficient. It is also one of the main reasons why targeting content without the discovery process of the value is far more likely to lose you revenue than generate more. It turns out the more you try to narrow a message or assume an outcome, the worse your results will be. Somewhat specific messages work for far larger groups than you could ever imagine, and you only know the true power when you let go of your own ego and preconceived notions and explore.

Stop thinking of personalization as trying to build a one on one message with a customer, that does not work and is extremely inefficient. Instead explore the various ways that you can create different content, and then explore who the largest groups are that you can present that to. This means always going through a discovery process of figuring out what matters, and then figuring out for whom. You may want to target to only people who looked at brand X, or page Y, or who have done a come to your site 3 times without purchasing, but that in no way means that you should limit the message to just that group. The less control you exert on the specifics of a message, and the more you are open to new possibilities, the more likely you are to find larger and more meaningful outcomes.

Explore what the value is of different messages and of taking it to different groups. You have powerful tools at your disposal to do just that, to discover and take these more general statements to large groups. From simple A/B tests all the way to automated machine learning, the real key to value comes from how you think about the problems, not in your ability to just find a group and target to it. Not only that, but you have the ability to measure the efficiency of various discoveries and techniques against each other. You are not limited to creating these stories, or just targeting to a specific persona, you have so much more at your disposal if you just allow yourself and others the flexibility to learn and grow.

P.T. Barnum is also famous for how he could get people to pay for anything, with the most famous example being the egress. It wasn’t meant for anyone specific, but he could get just about anyone to fall prey to the mystery. He didn’t have to target that message to just one group, or to offer it for only people who were on their way out, he figured out how to take that message to everyone. He understood that just because a group might be inclined for something, that just limiting your message to that group was a waste of his time. He was the ultimate salesman, but he knew that the key was to make it look like you were walking a fine line and being extremely specific, while at the same time in no way going that far.

So the question comes down, as you explore personalization, or you selling the egress? Or are you the one on your way out that door?

The Great Dark Truth of Analytics: Sociopath or Prisoner’s Delimma?

One of the great ironies of our industry is all the time wasted talking about “Big Data” or “Governance” or a thousand other small wasted catch phrases. All of these are simple attempts to show a “maturation” or growth of the online data world. While there is enough backlash to point out that none of this is really new, it is simply the online matching the offline Business Intelligence path. A world that never has produced 1/50th of the value it also pretends to. The irony lies not in what is being debated or not debated, or if online and offline are the same, but in the fact that we all talk around the real issue and do nothing to address it.

The reality is that most BI work is a wasted effort, designed to facilitate a predetermined wish of some executive, and that the people rewarded are never the ones that actually provide the best analysis, but the ones that produce the analysis that best helps make others agenda’s move forward. The online world is simply following suit, to the point that we no longer look at how many reports you can create as justification for existence, but now how can you create a fancy graphic or data to support one groups agenda or another. This evolution follows a normal path from creation and storage, access, and now delivery of data, without one time dealing with the real issues at hand. Human beings are both awful at understanding or leveraging data, but also most people (especially those in marketing) are awful at their jobs.

If you think about it, it’s not that shocking that marketers are horrendous at their jobs; they make a living telling stories and trying to convince others of things that have no basis in reality. This means to exist in this world, you are left with two options: Act like a sociopath, or unconsciously acquiesce to some sick permanent version of the prisoner’s dilemma, but in this version as long as no one points out how full of it the person speaking is, they will return the favor. Both diseases leave the same outcome, a group of people who exist to propagate the work of the same group of people, and who seek outside justification, be it awards (from the same group), data that is only searched for one way to support them, or case studies of people who did the same thing and likewise lied their way to “success”.

The scare and threat of data is that when used in a rational manner, it can shed light on the real value of the day to day actions we hang our hat on. We can find out just how horrendously off our preconceptions are. One of the great ways to succeed at testing is simply to bet against people, as people are so rarely right that just picking the other direction creates a winning streak that would allow you to live as a billionaire if you could translate it to Vegas. You will quickly find that most experts are nothing more then storytellers, and that most of the largest gains companies made are often the least publicized, but those that are shared are often subconscious attempts to get others to fall pray to the same mistake that they too wasted months on. With almost no effort you can prove that most actions taken provide no value whatsoever to the company, or are so inefficient that they are far worse. The data and evidence is easy to get, but we avoid it in order to cope within this world.

Why can’t people act rationally more often? Why is data accepted and abused, why do we seek confirmation not information? Why do we not worship those that get results instead of those that tell stories? The answer is simple, we fear most that which may hurt our own world view. If everyone was willing to search for the right answer, we would all be better off, but as soon as one weak person accepts the word of one sociopath, we are all set on down this path, or suffer silently the fight against the tide.

This is not a new problem, Kant, Engels and many others have been talking about this problem for hundreds of years; we just find new names for the same human weakness. We seek out people to do attribution, and then believe it tells you anything about generation. We seek out those that confirm our hypothesis, not those that disprove them, despite the fact that the disproven hypothesis inherently has a better outcome. We want people to speak at conferences and point out why everyone else is screwed up but why we can change and do the exact same things that they were railing on, but now with a new name. We want to find a site that tells us “which test won”, not one that helps us be better tests or to achieve any value whatsoever. We are constantly searching for the next affirmation to justify who we are, not improve who we are.

“Reality” is not a kind mistress for those that are even slightly interested in it. Empirical Realism is looked at and talked about, but practiced by so few that it is almost as meaningless a buzzword as “personalization”. While it helps companies, it rarely helps those that which to exist in a corporate environment (see prisoner’s dilemma). We are forced to make a Sophie’s choice, do what makes others happy or that which helps our company. We all try and find ways to convince ourselves and others that we are not faced with this choice, yet we only succeed when we stop caring about or thinking about anything other then our own gain. In order to facilitate this, we find every way possible to make the mental pain go away and to find others that will tell us it will all be ok.

So if this is the sandbox in which we play, is it any wonder that our “heroes” are those that best project the best ways to make others believe what is being done matters? We worship at the altar of Avinash, or Peterson, or Eisenberg’s or anyone else we can find that as justification for what we were already doing. We have no way of knowing if what they say is correct, and personal experience shows that following most of that advice leads to immensely fallible results. Far be it from an inquisitive mind to question if the current action is the right one, or if there is a better way to think about and tackle problems. We instead allow others to dictate to us, so that we can avoid cognitive dissonance and rest easy at night… ok, on second thought, most marketers are both suffering from living in a prisoner’s dilemma and are also sociopaths. The data shows that these are not mutually exclusive but complimentary. Glad we got that squared away…

If you want to really make a difference, if you are tired of this same old world or of those charlatans who propagate it, are you prepared to fight the tide? Are you able to evaluate your own work, to go past the comfort and to find out how wrong you are, in just about everything you do? Are you then able to get past that mental scarring to do the same for others? Will you back down the first time someone pushes back, or will you make it your quest to do the right thing when it is neither profitable or easy to do so?

The history of business shows that rarely if ever does this problem truly go away, or does the better answer win. While history is written by the winners to justify their existence, randomness and the trampling of others are bred into every page of this twisted form of storytelling. And yet, until we deal with this real problem, until we are more interested in doing the right thing then the easy one, what will really change?

We will continue to waste time and effort on data, in order to justify wasted time and effort in most other efforts. We will continue to seek new words for old problems, and we will continue to make heroes those that most hold us back. Until we stop propagating the lie and until we look at ourselves first, how can we ever really deal with the real problem of data. Not the collection, not the sharing, not the presentation, but the people who are wired to use that data in the least efficient and most self-serving way possible. You want to solve big data, you want to change the industry, stop wasting time on tag management or Hadoop, solve the people, since there is where all problems lie. Don’t solve how do you share your point, but how do they think about and are they rationally using data to find an answer, or only to justify one?

The Road to Greatness: The Do’s and Don’ts of Building a Proper Testing Infrastructure

There is a lot of misunderstanding on the difficulty of using Adobe Target. There are all sorts of uses of the tool, and in a lot of cases, people confuse their inability to set-up a proper infrastructure with limitations of the tool. The wonderful thing about Target is that it has 5000 different uses. The difficult thing about Target is that it has 5000 different uses. Figuring out what to do, when to use it, and how to tackle that problem is paramount to a mature program.

Before we get to specific suggestion however, I want to make it clear what I consider a proper technical infrastructure. It comes down to how you answer this question? Can you get 80% of all tests live, with the proper amount of variations, live in 30 min or less. That is from concept to launch. That number seems crazy to people at first, but the reality is that you can get a test fully live in 5 min if you have everything ready to go. That is the sign of a successful infrastructure: speed of execution, understanding and flexibility. In order to do that, you have to have things set-up in a way that will make you successful. Here are the key things to do and avoid in that quest for speed and execution without sacrificing (in fact improving) results.

Do – Make sure that you can test on all major part of your site today

Make sure that you have mboxes on all key parts of your site. You will most likely want to make sure that you have multiple mboxes on those key pages, usually leveraging a mixture of global and wrapped mboxes throughout the page. Global mboxes are ones that do not wrap any content, but exist at the very top of the page to control CSS, redirects, and other functions. Wrapped mboxes are ones that wrap key elements of the page, with the best practice of never wrapping more then 1/3rd of the viewable page with a single mbox. The key here is to make sure you are not adding and removing mboxes for each test. One of the nice things is that even if you do have those mboxes on the page, you can disable them easily in the console so that you are not charged for them.

Don’t – Have more then 5-6 mboxes on any page

Just because you want to prepare a page to succeed does not meaning going crazy with mboxes. The reality is that each request does impact site performance, and any noticeable interaction with the page will result in the validity of a test result being highly questionable. Setting up a page to be able to do 80% of the things you can do is important, just as prioritizing those efforts to leverage what you can do today to learn about the other 20% that require additional resources. If you can do 80% of possible things today, then do those things, and you will find that often the tests you are not sure about are the ones that produce the biggest and most meaningful winners.

Do – Make sure that you are tracking key information about your users

This is both an on-page and in console element and may require a bit more time and discipline to think about before you act on anything. This means leveraging one of the most under-utilized but extremely valuable parts of the tool in the profile system. Make sure that you are passing key information you might have from your CMS or other systems about your users where necessary, and also make sure that you are recording key pieces of information in the profile system. The key here is to look into it, but not believe it blindly, to search for value and not predetermine outcomes. Just because you really want to target to a certain membership level does not mean you should ever target to them before you test and measure the value of that change compared to other alternatives. Even if you aren’t leveraging it today, it can be leveraged in the future, and it allows you to easily add that information for analysis later.

Don’t – Delay your infrastructure for unnecessary pieces of information

Most groups tackle optimization as just the end function of a number of different inputs. The worst examples of this is when groups go through and interview for a list of all possible pieces of data a group may want to look at or segment information by. While it might make your life easier since you don’t have to correct anyone, the reality is that you are most likely causing massive damage to your program. It is a good idea to get a read on what people may want, but the primary directive should be to help groups think and act differently with data. As part of that, you need to ask the fundamental question of “will the data I am collecting improve our business enough to be worth it, and even so, can I get better outcomes from using those same resources?” There will be cases when the answer is yes, but in most cases if you are honest the answer will be a resounding no. In those cases, do not delay other actions just to make others happy. Use what you have to get the most with what you got, don’t waste time waiting for everything to be perfect.

Do – Make sure that you can get tests live without IT involvement.

You should be able to do 80% of tests with some very basic understanding of how a webpage works. After you get your infrastructure in place, you do not need IT support except in one off cases. You should not need IT except I extreme circumstances and only then once you have proven via discovery that the item you are using those resources on is the most efficient use of everyone’s time. Where groups run into major problems is when they view each test as a technical project, which gives both the wrong impression about the difficulty of testing and makes things much slower to react then they should be. Even worse are groups that think that technically complex means valuable, which is almost always the opposite of the truth. Difficult means difficult, valuable means valuable, those two things do not have much to do with the other.

Prioritize tests by how much effort they are to get live, as well as how many different feasible alternatives you will get, and how much they challenge assumptions. Do that, and you will dramatically improve the outcomes and reduce the resources needed for your program.

Don’t – QA testing like you do everything else on your site

There is nothing worse for testing then having a test ready to go and then having to wait weeks for it to go through a standard QA process. Make sure you do a good job of QA, but keep in mind that nothing will ever be perfect, and that you can get most QA done for most tests by simply passing around a few QA links to a couple of colleagues and having everyone use a couple of browsers. You will most likely want to do more QA for efforts that dramatically change site function, but the reality is that you should be fewer of those tests and far more of the more basic tests anyways. Add better rights control so that fewer people can then push things fully live also adds meaningful limits to make sure you are accomplishing what is needed without sacrificing speed and efficiency.

The promise of the tool is that marketers start having control of their site, so why then do people so willingly give up that control? You need to be able to understand why you need to QA differently, and how different types of tests impact site function, so that you can have a meaningful conversation with yoru tech team and help them understand that you are not going to be constantly breaking their site. Be thorough, but don’t go crazy or just dump the responsibility onto others.

Do – Make sure that everyone is aware of when you are running a test

One of the benefits of setting up your campaigns to use QA parameters is that those links can be shared with everyone and you can therefor make more people aware of a campaign. There is nothing worse than having a campaign live for a few days and then having to stop it because some VP randomly hit a test variant and thinks the site is broken. Communicating both launch and results and especially lessons learned across tests is also part of a successful infrastructure.

This will also help build awareness and interest in the results of the tests. If you design tests with enough variants and challenge enough assumptions, in almost all cases you will get results that make other groups fundamentally challenge their own ideas of what works.

Don’t – Ever launch a test without a clear idea of how you are going to act on the data

There is no point in any test if you are not clear on how you are going to act. This means knowing your single success metric, but also what happens to push a winner, or how you are going to follow-up, or what you will do with segment information. Having those conversations outside of the specifics of a test and having groups aware of their roles before the launch is vital to a speedy and successful infrastructure.

Being prepared to act and having your groups ready to act quickly and efficiently can many times be against the entire history of some organizations. This is why it becomes so vital to build a proper infrastructure on every level to enable testing to work. Focusing on what matters, instead of just how to get a single test live, is one of the key differences between groups who just test and those that run successful testing programs. Make sure that you focus on the things outside of a specific test and help move the various groups involved towards an end goal, and you will be amazed at how fast you will act and the results you will achieve.

The Road to Greatness: The Do’s and Don’ts of Starting an Optimization Program

As more and more programs start to emerge with the growth of the online optimization field, there becomes a preponderance of “best practices” when it comes to testing, personalization, and all other active forms of leveraging data, it seems like you have to know a massive amount to just understand completely what those “experts” are saying.

With that in mind, I wanted to present some very simple do’s and don’ts for programs just getting going. Starting correctly and setting the stage for success is vital to really being efficient and getting value out of your program that you can and should. The problem is that in almost all cases people’s first instincts lead them astray. What you don’t do is more important usually then what you do choose to do. The key is to make sure that you focus your limited time on the actions that will provide the greatest growth and value to your program. The same advice can work for groups that have been testing for years as many of those programs also are just built up versions of the same bad behaviors.

DO – Hold discussions about a single success metric

The very first and sometimes the most painful hurdle that a program faces is getting groups to agree on how to act. This is in many ways completely counter culture as many groups have competing goals and are only focused on their little piece of the larger pie. If you do nothing else, getting people to agree on the one thing that you can all make a decision on is vital.

A side benefit of this conversation is that it starts the process of allowing people to dissociate the actions they think will lead to success and the actual measure of success. Way too many people think that if their idea is more people looking at product X will generate additional revenue then the measure of success is more people looking at product X. You may have an idea for what you want to do, but you are doing it to accomplish a goal, so measure the goal, not the action. The measure of success would be additional revenue, and once that is the only goal, you can start comparing all feasible ways to achieve that exact goal.

DON’T – Get too caught up on test ideas

Some of the least important parts of a testing program is the generation of test ideas. While this is the fun part for people to try and prove their point, the keys to success are not in having a bunch of ideas, but in putting together the infrastructure and helping understand the discipline of successful testing. Test ideas will come naturally out of everyday conversations and especially out of prior tests and learned knowledge. There is never a lack of things you can do, but focusing too much on that part allows people to get caught up in many different biases which will make their rational evaluation of the results to collide with their ego.

DO – Apply tech resources on a larger infrastructure

All tools require some sort of deployment, and while some are easier than others, the biggest mistake you can do is to think that every test will require massive amount of resources. If you build a proper infrastructure across your site, then most tests will not require any involvement from development resources whatsoever.

The key to a good infrastructure is to have tagging in the key locations on your top pages so that you can test just about anything. You will also need to make sure that you have tracking in place for your success metric, and for any additional information (like segment information) that you may want to provide.

Testing should not be thought of as a project but as an ongoing organization and site feature. It is something that should be set-up in a way to never stop and to never be about the simple validation of a single idea. In order to maximize this, going through the initial “pain” of a larger deployment and making sure that your IT group understands that this is not a permanent engineering owned project will dramatically improve your ability to move quickly later on. The key once this is done is to prioritize tests based on resource usage and prioritize tests that will deliver the greatest return for the lowest resource usage.

DON’T – Think testing is just an extension of your analytics group

How you think about optimization is almost the exact opposite of analytics. Instead of patterns and anomalies of larger data sets, you have a single point and the push to make consistent meaningful changes. Testing is not just the action arm of some analysis you did to validate your point, it is the active acquisition and interaction with data.

To succeed, you need to think about segmentation differently. You need to think about what a success metric really is and how it is different in testing. You need to able to speak in terms of comparative analysis, not validation. Basically, you have to be able to turn just about everything you do with analytics on its side. Later on, you can start leveraging the two together, but as you start, separating them completely is going to grant you far more return with far less work then trying to just tack testing into your analytics daily activities.

DO – Think about your rights management

Make sure you know who is going to have what rights and make sure you have some checks in place from too many people changing your site.

DON’T – Blindly follow statistical measures

You don’t need to know everything about all statistics, but you do need to understand some basic concepts to really understand results. The first is that for any statistical tool to be useful, you need not just statistical confidence, but you also need the data to be representative of the change you are going to make. If you get 99% confidence in 3 hours on a Friday afternoon, that data is only representative of that period of Friday afternoon.

DO – Starting thinking how you are going to store and share results

When you are testing right, you are going to constantly learn new things and if you are doing your testing right these lessons you learn will eventually be far more valuable than any individual result. You need to start thinking about where you are going to share this information, the format, and the availability. You also need to make sure that this is not a static item but a living knowledge base.

DON’T – Let any test go out with just two recipes

One of the hardest lessons to learn is that testing is not about validation of a single point, but about comparing feasible alternatives and being prepared to go in directions that you never imagined. While not everyone will be ready for this day one, the simplest way to prepare people is to force discipline on them. Making people have multiple very different but feasible alternatives will start giving you far more information and will start to show you areas where what they thought mattered didn’t.

There are a thousand other things that go into running a program, but just starting out, if you tackle these simple things and avoid some common traps, and then you will be get far more results, make a larger impact to your business, and use far less resources. Think about what you really want from your program and then stop focusing on individual tasks and instead start putting the key pieces in place for long term success.