Category: Rants

Rant – Why whichtestwon Makes you a Worse Tester

There is nothing less important then what the winning recipe was of a test.

I want to let that sink in.

Everyone loves to get caught up on which recipe won, because it is what you look at and it is what others want to know, but as a tester, it is the way that you arrive a that you arrive at that answer that determines if you actually provide value or just an answer. Individual outcomes interest people who have something invested in being “right” where consistent meaningful discipline is what matters for people who are invested in improving things consistently. If you only discovered something that is the 2nd best out of 10 different feasible alternatives, you wouldn’t pick the 2nd best, but when you only compare two things, that is most like what you are doing. You haven’t accomplished anything and you are actually losing money. If you didn’t actually measure outcomes of multiple alternatives, or if you didn’t measure against a global site wide metric, or if you did not account for the cost to arrive at that conclusion, then you are fooling yourself into thinking you have accomplished something when all you did was take resources from others to make yourself look good. It may impress others, but it has not provided one bit of value to the organization.

In order to be the best alternative, you need context of the site, the resources, the upkeep and the measure of effectiveness against each other. Even is something is better, without insight into what other alternatives would do it is simply replicating the worst biases that plague the human mind. Figuring out the better of two options is an answer, finding out the value of different feasible alternatives is providing value. Finding out who was right “picking the winner” is great for people’s ego, but making sure you are measuring multiple alternatives and that you are choosing the options that provide the highest return to the largest population for the lowest cost is what makes you successful.

To make it worse, people then look at the results and think that they will get the same result for their site, and in the worst case, they do. Sites like whichtestwon, which focus on letting people find out what won amongst two options sound great, and capture people’s attention. They let you guess and pat yourself on the back when you are right or wrong, but the reality is that they are designed to feel good but not actually provide value. If you wanted a site like that to provide value, then they would require

The problems of a tester are two fold, one in convincing others to test, and second in improving the testing to make sure that you are maximizing return and lowering cost. A good tester needs to be able to balance both, since there is little to gain outside of personal reward in just foolishly running tests. But sites like whichtestwon? It is designed to assist the first; to provide evidence for people that you can get an positive outcome (missing that you also get outcomes from other uses of the same resources) without actually giving any real insight into if you did provide a positive outcome (an outcome, by itself, tells you nothing). It is designed exclusively for people to abuse to push their own agenda. To take a quote directly from their tour:

Site shows stats from various A/B tests – Finally I’ve got evidence to show clients on a load of design decisions!”

That shows everything that is wrong. Testing should be about seeing what the value of different test variants are, not making the case for a specific one that you want. In order to be successful, you have to prove yourself wrong. If it would have worked the first time, then there was no point in the test (and you are wasting resources to run the test) and you have learned nothing. You should not be given “credit” when you are adding additional cost and providing nothing more then validation for others. When you are wrong, when you have tested what you want and tested other alternatives, and you find other alternatives prove to be more efficient, even if what you wanted was better than control, that is the moment you are truly gaining something from your testing efforts.

There is a plague of people in our industry who try everything that can to show how much value they got from a single test. Who view testing as a way to get what they want up on the site over the HiPPo or someone else. Who abuse testing to push their agenda and who then take credit when they find something that proves better then what was there before. The act of running a test is not a measure of success, nor is having an outcome. Added value only comes from finding an outcome that is different then what you would have already done. In order to do that, you must measure multiple feasible alternatives and find an outcome different then what people want. If you aren’t able to do so, then the most fundamental problem you have is you, and how you think about testing. If you are able to, then the individual outcome, what won, is far less important than how you got there and what you chose not to do. The measure of a testing program is how often they are proving people wrong, and about how consistently you can do that with the least amount of resources possible.

Being a good tester means that you always know the relative costs. It means that you know how often something works, not just if it did one time. To be good, you should be able to create meaningful actionable lift on all your tests, not jump up for joy and promote yourself to the world when you managed to find one thing better on 1 out of 5 tests. Don’t settle for taking the easy road and trying to take credit. Add value, be better, learn how to look at things and you will actually create value, today and always. If you go down that road however, then no one cares which variant won, it has no bearing on long term success. Great, you found the thing to push from this campaign, that is just one small step on a long road of continuous action. You wouldn’t reward someone because they managed to turn write their name on a test, so please do not think that whichtestwon somehow does anything to inform you how to be a better tester.

If you really wanted to see a site like whichtestwon matter, then show the variants that didn’t win. Show multiple options for each outcome and show what the best option was? Give us a measure of the cost and give us the internal roadblocks that you had to overcome. Let us know if that outcome was greater or worse then others for that group and what they are doing with the results to get a better more efficient result next time. If you are interested in anything more than self-promotion, post the things that don’t work. Tell us how often something wins, not the one time it did win. Use the site to find examples of where you were wrong and inform yourself that you are not right… ever. The most we can ever hope to be is a little less wrong and working on a way to speed up the process for discovering just how wrong we are.

Are You the Hero or the Villain?

Thanks to Brent Dykes, there has been a lot of talk recently about analytics action heroes. Everyone wants to be a hero, and everyone thinks that they are one or on the road to being one. My work unfortunately has me often facing the opposite; programs that are not succeeding, often due to villains. One of the great truths is that the villains never know that they are the villains, often thinking they are the real hero. They are constantly talking about action, they are involved, and more then anything they speak up for the use of data in the organization. To be a real villain, they have to be capable and smart, just like a hero, otherwise the damage they do would be mitigated. The problem is that they do it for all the wrong reasons and without the goal of actually improving performance. No one wants to be the villain, but why then do they so outnumber the heroes in our industry?

So how then do you know if you are the villain or the hero?

There is no magical litmus test to get your hero card, but there are many common traits that define the members of both groups. Here are a few barometers that might help you define where you are and what you need to work on to be whichever role you are trying to be.

Position –

There are heroes and villains at every level. It is not always a HiPPO versus the low man on the totem pole. Analysts and marketing managers are just as likely to run a program into the ground as VPs and CMOs. It isn’t about your title but about what the actions you take towards the program. Are you talking about making a difference while choosing actions that make you look good? Or are you actually doing the small things that aren’t looked at that really make a difference?

Heroes view their role as finding the best answer and doing what is needed to make the site succeed. Heroes judge their position by what they do to make others better. Villains view their roles as doing what their boss wants or what will make them look best. Villains use the position to focus on themselves. Heroes are interested in ignoring their “title” to do what is needed. Villains use their title to take credit for things and to keep things under their empire. Heroes know that there are many hurdles, but they won’t accept excuses. Villains are the first to complain about others, but then accept problems as excuses and then spend a great deal of time reminding you why it is the other person’s fault. Heroes know that you don’t know the answer to everything and that discovery is part of excellence. Villains tell everyone they have the answer and then find data to support their position and make them look better. Both sides talk about trying to do what is best, but the actions and the excuses determine quickly which side of the battle someone falls on. Everyone claims to do what is best for the site, but actions speak louder than words, and if you are worried about keeping people happy or doing only what you are told, then you are not doing what is best for the site.

Skills –

Heroes’ skills are in finding multiple answers to problems and figuring out the efficiency and the value of each one. Their skills help educate people about what defines a good answer. They are capable of giving a presentation, but they are at their best with changing people’s misconceptions and finding the best answer, not just the first one that comes up. They know that to be successful, they need to know a little bit about everything and they never accept “I don’t know” as an acceptable answer. They go beyond what is asked and never settle for “best practices” or just returning a report. They know that just because their boss wants an answer to question A, that the company might be better served finding the answers to the questions that aren’t asked, so they focus their skills on finding those questions and answering them, even if that is not supporting someone’s agenda.

A villains’ primary use of their skill is directed towards self-promotion. They take every opportunity to show how valuable their “contributions” over focusing on what real value of the actions taken. They view their job as improving their “personal brand” and are more than happy to find data to support others claims or agenda, as opposed to finding the best answer. They are the first to dive in and find the answer to the questions their bosses are asking, even if that question has no real value. They blame others when they don’t know something and they are more than happy to tell others it’s their job to “figure it out”. They spend their time focusing on improving their presentation skills, networking, and self-promotion skills. All they want is to find an answer to the requests before them to make the people above them happy. They find no reason to find more than one answer or to challenge ideas, because the act of finding that answer makes others happy and helps them show their “value”.

Research/Community –

Heroes love to research and view the thoughts of others. They do not however look at only one community or think that just because someone gives a great presentation that they are correct. They appreciate popularity, but know that the more people read a blog or buy a book, the more likely the material is to be what people want to hear and not actually valuable content. They don’t just accept a statement from anyone, especially when it sounds like exactly what they want to hear. They view the world through a lens trying to find everything that can be fixed and what is wrong with the current process. They don’t make excuses about time to dive into multiple disciplines or to find the latest news. They know that the time used to find a better way to do things will make them have multiples of that time available later. They take the time to read and find the best and worst quality materials out there because they care about content and know that simple almost never equals right. They know that you need lots of different perspectives on a problem to understand it, and that there is no single answer to any problem. They understand that today’s answers will prove to be wrong tomorrow, so they aren’t concerned with trying to prove themselves right as much as they are in finding the next “best” answer. They search out new perspectives and new people to continue a search for improvement.

Villains are also heavily involved in communities, in fact some of the most vocal and famous part of communities are villains. They use research and communities to promote their image and to tell the world how great they are. They find new ways to say the things that have already been said and view their self-worth and value as the act itself of making a presentation, not in the value of the content shared. They love to build their own groups in those communities in order to have more people propagate whatever myth they are selling at the moment. They are also always searching for the next big thing in order to get ahead of it, tell the world how they mastered it, and also to move on from what they were doing before the reality of their failure becomes evident. They don’t research or use community to find what is wrong with what they are doing, but instead to validate and promote their own agenda. They try to find what they can from every piece of information in order to make themselves look better and to bring others under their political umbrella.

Technology –

Heroes view technology as a means to an ends, one that is often foolishly rushed into to meet someone’s agenda. There are great technologies out there, and no one would be able really achieve anything if it wasn’t for the great technologies in our industry, but they focus on getting things right, building out the right disciplines, the infrastructure, and not just learning one way but the best way to leverage any a tool to do a predisposed function. They aren’t impressed with having 50 tools running on your site, but instead with how many you have running in a way to really improve things. They live by the creed, “You can fail with any product” so they focus on creating the infrastructure to make the products they do have succeed. they know that just being able to collect data does not magically make it valuable. In order to do this, they are aware of all the various offerings on the market, but focus on the efficiency of each one. A hero is more interested in how often things go wrong and how to make sure they don’t fall into that trap then worrying about the latest great “success story“. They aren’t afraid to challenge sales pitches and “experts” to find the best answer.

Villains make their career on buying and getting the latest technologies. They love to be able to promise the next great thing internally and to “own” it to help themselves look good. They don’t care about what the likelihood of success is, but instead what they can sell internally about the “value” they are being promised. They rush to evaluate and get as many new technologies and to stay “ahead” of the field. They aren’t interested in building an infrastructure for success, but instead focus on what promises they can get to promote themselves internally. They spend their time “evangelizing” and not getting better. When things don’t work, they move on to the next technology or the newest industry buzz word and find someone to take the blame. They don’t care about building a successful program as much as they care about “integrating” all these technologies and finding a story to show their boss.

There are hundreds of other comparisons you can make between heroes and villains. The truth is that we are always having to balance one side versus the other. It may seem like a fine line between hero or villain, but remember that it is always up to you what actions you choose. Heroes know that you are forced to choose between doing those actions that make you look good and the ones that make an organization successful. Heroes accept the sacrifices and don’t make excuses. Villains convince themselves that they are the same thing and that what they are doing in all cases makes the organization better. No organizational structure or mental evolution of a program will make up for having villains in your program. We all talk about doing the right things, but at the end of the day, it’s not the stories your tell others or the justifications that you make to yourself, but your actions that determine which path you take.

The real question for you is, which do you want to be and if so, what are you doing to get there? All heroes have to go through a quest to earn their abilities, often with many hurdles and defeats. They are often not immediately rewarded for their skills and misunderstood, but in the end, they emerge victorious. There are always hurdles before you and you are always going to be searching for a way to get past them.

When your story is told by others, are you the hero or the villain?

Rant – Door #1 or Door #2?

There is a moment in every engagement where you have to make the choice between making the client happy or making them successful. There is always something (and usually many many things) that they want to do, that will lead them down a very dangerous path. Those same things are almost always their babies, tied to things like personalization, or improving metrics that make them look good, or a hundred other things.

When that moment comes, it is amazing how often people choose the “make them happy” path. Most do it by not acknowledging the issues, other make excuses to themselves that they will deal with it later, or that the client was going to do it anyways. Most make sure they are in a state of plausible deniability by not knowing what the value of things are or by believing the so called “experts” of their field, most of which know less then a low level analyst, but who focus on telling people what they want to hear and by being good speakers (There is no correlation between being a great speaker and having great ideas”.

One of my favorite managers used to have a saying. Whenever you didn’t do what you were supposed to, he would ask you “door #1 or door #2?”. This was a reference to lets make a deal, where door #3 is the prize (you did the right thing), and behind the other two doors, you got the donkey. In this case, Door #1 refers to the fact that you are incompetent – You tried but weren’t up to the job. Door #2 refers to you giving me the middle finger. You could have tried more, or could have done more, but you chose not to.

When you make that choice, when you choose to improve your relationship, or you choose to make them happy, over making them successful, which door is it? What is so important that you can’t do the right thing. It isn’t easy, and the truth is that in most cases, the client won’t ever know that they aren’t getting anywhere near the value they should. But you SHOULD know, and not knowing, that is door #2. If you do know, and you try, and fail, then it is door #1. The choice to not fight the battles that are worth fighting, or to promote yourself over doing the right thing, that says more about you then any engagement or any situation.

So which is it?

Door #1 or Door #2?

Rant – More Tests Do Not Equal More Value

I continue to see the propagation of one of the worst myths in our industry. The way to get more value from your testing program is to run more tests. That is just lazy speak for spend more money with us or give me more money if you want real results. That is a sign that you haven’t spent the time to learn how to test, and are just propagating the myths of the industry and using it to make yourself look good.

The key is to be efficient and to design your tests in a way to insure that you CAN NOT HAVE A FAILURE. I am dead serious that I can say that I have not run a test in 2+ years now that has failed to deliver both insight and meaningful actionable results that make money. In almost all cases, it took doing things that took LESS resources then what the people I was working with wanted to do. It is not magic, it is hard work, constantly finding better ways, challenging assumptions and doing everything I can every day to prove myself wrong and to find a better way. This is not about me, I have shown many people that care how to do the same, and some of them have gone on to make what I was doing at the time look foolish. I love those moments and appreciate every time they happen.

Its only when you are lazy and doing “better” testing that it is even possible to have a failed test. If you are not seeing value, change how you do things, don’t just keep throwing more resources at it until you find a story to tell your boss. So many people are afraid to challenge the thoughts of others, or to find out how wrong they are, and because of that, they just use whatever tool is fashionable to keep doing what they were already doing. If that is what you are doing, then it doesn’t matter if you do get positive lift, you are still losing almost all your money. It is not about the number of actions, but the value of the action. Doing more low value things is just throwing away money. The entire point is to figure out efficiency and to put resources in the most efficient way possible. Actions are not random, and just blaming others and resources is not the answer.

Fix what you are doing. Learn, get better, learn what about the system and stop trying to force it into some preconceived notion. You may get more if you have more time/resources, but that is not an excuse for not using what you have in the most efficient way possible. Stop making excuses, stop looking for outside reasons for your failures.

Your success and failure comes from the actions you choose or don’t choose to do. It isn’t magic, it isn’t a great conspiracy from others against you. It is your actions that get results. Stop trying to do more with more, do more with less or the same and you will find much better resources.