Are You the Hero or the Villain?

Thanks to Brent Dykes, there has been a lot of talk recently about analytics action heroes. Everyone wants to be a hero, and everyone thinks that they are one or on the road to being one. My work unfortunately has me often facing the opposite; programs that are not succeeding, often due to villains. One of the great truths is that the villains never know that they are the villains, often thinking they are the real hero. They are constantly talking about action, they are involved, and more then anything they speak up for the use of data in the organization. To be a real villain, they have to be capable and smart, just like a hero, otherwise the damage they do would be mitigated. The problem is that they do it for all the wrong reasons and without the goal of actually improving performance. No one wants to be the villain, but why then do they so outnumber the heroes in our industry?

So how then do you know if you are the villain or the hero?

There is no magical litmus test to get your hero card, but there are many common traits that define the members of both groups. Here are a few barometers that might help you define where you are and what you need to work on to be whichever role you are trying to be.

Position –

There are heroes and villains at every level. It is not always a HiPPO versus the low man on the totem pole. Analysts and marketing managers are just as likely to run a program into the ground as VPs and CMOs. It isn’t about your title but about what the actions you take towards the program. Are you talking about making a difference while choosing actions that make you look good? Or are you actually doing the small things that aren’t looked at that really make a difference?

Heroes view their role as finding the best answer and doing what is needed to make the site succeed. Heroes judge their position by what they do to make others better. Villains view their roles as doing what their boss wants or what will make them look best. Villains use the position to focus on themselves. Heroes are interested in ignoring their “title” to do what is needed. Villains use their title to take credit for things and to keep things under their empire. Heroes know that there are many hurdles, but they won’t accept excuses. Villains are the first to complain about others, but then accept problems as excuses and then spend a great deal of time reminding you why it is the other person’s fault. Heroes know that you don’t know the answer to everything and that discovery is part of excellence. Villains tell everyone they have the answer and then find data to support their position and make them look better. Both sides talk about trying to do what is best, but the actions and the excuses determine quickly which side of the battle someone falls on. Everyone claims to do what is best for the site, but actions speak louder than words, and if you are worried about keeping people happy or doing only what you are told, then you are not doing what is best for the site.

Skills –

Heroes’ skills are in finding multiple answers to problems and figuring out the efficiency and the value of each one. Their skills help educate people about what defines a good answer. They are capable of giving a presentation, but they are at their best with changing people’s misconceptions and finding the best answer, not just the first one that comes up. They know that to be successful, they need to know a little bit about everything and they never accept “I don’t know” as an acceptable answer. They go beyond what is asked and never settle for “best practices” or just returning a report. They know that just because their boss wants an answer to question A, that the company might be better served finding the answers to the questions that aren’t asked, so they focus their skills on finding those questions and answering them, even if that is not supporting someone’s agenda.

A villains’ primary use of their skill is directed towards self-promotion. They take every opportunity to show how valuable their “contributions” over focusing on what real value of the actions taken. They view their job as improving their “personal brand” and are more than happy to find data to support others claims or agenda, as opposed to finding the best answer. They are the first to dive in and find the answer to the questions their bosses are asking, even if that question has no real value. They blame others when they don’t know something and they are more than happy to tell others it’s their job to “figure it out”. They spend their time focusing on improving their presentation skills, networking, and self-promotion skills. All they want is to find an answer to the requests before them to make the people above them happy. They find no reason to find more than one answer or to challenge ideas, because the act of finding that answer makes others happy and helps them show their “value”.

Research/Community –

Heroes love to research and view the thoughts of others. They do not however look at only one community or think that just because someone gives a great presentation that they are correct. They appreciate popularity, but know that the more people read a blog or buy a book, the more likely the material is to be what people want to hear and not actually valuable content. They don’t just accept a statement from anyone, especially when it sounds like exactly what they want to hear. They view the world through a lens trying to find everything that can be fixed and what is wrong with the current process. They don’t make excuses about time to dive into multiple disciplines or to find the latest news. They know that the time used to find a better way to do things will make them have multiples of that time available later. They take the time to read and find the best and worst quality materials out there because they care about content and know that simple almost never equals right. They know that you need lots of different perspectives on a problem to understand it, and that there is no single answer to any problem. They understand that today’s answers will prove to be wrong tomorrow, so they aren’t concerned with trying to prove themselves right as much as they are in finding the next “best” answer. They search out new perspectives and new people to continue a search for improvement.

Villains are also heavily involved in communities, in fact some of the most vocal and famous part of communities are villains. They use research and communities to promote their image and to tell the world how great they are. They find new ways to say the things that have already been said and view their self-worth and value as the act itself of making a presentation, not in the value of the content shared. They love to build their own groups in those communities in order to have more people propagate whatever myth they are selling at the moment. They are also always searching for the next big thing in order to get ahead of it, tell the world how they mastered it, and also to move on from what they were doing before the reality of their failure becomes evident. They don’t research or use community to find what is wrong with what they are doing, but instead to validate and promote their own agenda. They try to find what they can from every piece of information in order to make themselves look better and to bring others under their political umbrella.

Technology –

Heroes view technology as a means to an ends, one that is often foolishly rushed into to meet someone’s agenda. There are great technologies out there, and no one would be able really achieve anything if it wasn’t for the great technologies in our industry, but they focus on getting things right, building out the right disciplines, the infrastructure, and not just learning one way but the best way to leverage any a tool to do a predisposed function. They aren’t impressed with having 50 tools running on your site, but instead with how many you have running in a way to really improve things. They live by the creed, “You can fail with any product” so they focus on creating the infrastructure to make the products they do have succeed. they know that just being able to collect data does not magically make it valuable. In order to do this, they are aware of all the various offerings on the market, but focus on the efficiency of each one. A hero is more interested in how often things go wrong and how to make sure they don’t fall into that trap then worrying about the latest great “success story“. They aren’t afraid to challenge sales pitches and “experts” to find the best answer.

Villains make their career on buying and getting the latest technologies. They love to be able to promise the next great thing internally and to “own” it to help themselves look good. They don’t care about what the likelihood of success is, but instead what they can sell internally about the “value” they are being promised. They rush to evaluate and get as many new technologies and to stay “ahead” of the field. They aren’t interested in building an infrastructure for success, but instead focus on what promises they can get to promote themselves internally. They spend their time “evangelizing” and not getting better. When things don’t work, they move on to the next technology or the newest industry buzz word and find someone to take the blame. They don’t care about building a successful program as much as they care about “integrating” all these technologies and finding a story to show their boss.

There are hundreds of other comparisons you can make between heroes and villains. The truth is that we are always having to balance one side versus the other. It may seem like a fine line between hero or villain, but remember that it is always up to you what actions you choose. Heroes know that you are forced to choose between doing those actions that make you look good and the ones that make an organization successful. Heroes accept the sacrifices and don’t make excuses. Villains convince themselves that they are the same thing and that what they are doing in all cases makes the organization better. No organizational structure or mental evolution of a program will make up for having villains in your program. We all talk about doing the right things, but at the end of the day, it’s not the stories your tell others or the justifications that you make to yourself, but your actions that determine which path you take.

The real question for you is, which do you want to be and if so, what are you doing to get there? All heroes have to go through a quest to earn their abilities, often with many hurdles and defeats. They are often not immediately rewarded for their skills and misunderstood, but in the end, they emerge victorious. There are always hurdles before you and you are always going to be searching for a way to get past them.

When your story is told by others, are you the hero or the villain?

Optimizing the Organization: Where Does Testing Fit into your Org Structure?

Where then does testing fit into your organization? Is it just something people do? Is it a central component to be shared, or is it just something that each group does on their own. All groups face this struggle when they discover that you can’t just make it an additional duty for someone and get good value. When you have decided that you want to build a real optimization team within your organization, you are challenged with those very questions. What I want to propose are a couple of frameworks that I have seen work to great success, and what fundamentally makes them successful. It may not be possible to move mountains quickly to get to these structures immediately, but it is important to understand why they work and to think of ways to move towards those directions.

One of the core challenges is that testing really isn’t a full on marketing discipline, nor an analytics discipline, IT discipline or really anything else that is in most groups normal organization structure. To most groups starting out, testing is thought of as a feature done in order to prove that their current efforts are better then their prior efforts. Testing instead is a unique discipline that takes parts of all of those, but is also at its best when it is showing the inefficiencies in your internal processes and mindset. for any group to succeed, you need to have people, alignment, and the correct mindset, otherwise it all goes to waste. When testing is not allowed to be a new discipline, it suffers exponentially the inefficiencies of the discipline it is placed under.

The first thing to understand is that for larger organizations, testing works best in a hub and spoke model; meaning that you have a central team that then works with members of the various business units to improve their actions. While this might be the best model, it only works if you have established clear rules and the correct mindset in those other groups as well as your own.

Education will always be the primary role of the testing team.

Each of the frameworks below shows the central team, that would then work with an individual or team in each structure that follows a similar format. Central alignment allows you to separate the resources and to insure that you are not just adding on testing to existing duties. This format allows you the benefit of creating a central knowledge base, while leveraging local knowledge, resources, and structure to work. The accumulation and sharing of knowledge, and the design of your efforts to accomplish this task is the primary goal of your larger structure. For this to work, testing can not be dictated only by the business unit. It must instead be a collaborative effort where both sides work together to create a continuous culture of optimization, one that is focused on being “wrong” over being “right”. There can never be a time where just someone coming up with an “idea” is allowed to be the end all of what is tested. No idea by itself is sacred, and no one, the CEO down to your janitor, should be allowed to just throw something up because they think “it will work”.

Framework #1:

In this first model, you see that we have a manager of testing who works through or acts as a project manager for external resources (IT and creative). This person may have analysts working under them, but fundamentally they work with other groups in a dotted line creating a cross functional team that tackles testing. While you may not have direct full time people on the team, in this structure the same people work together regularly to advance the organizations optimization efforts. You will also notice that while they may work under analytics, they are not analytics, with a separate team handling those responsibilities. The disciplines are dramatically different, and there is a lot of value for bringing different data types together, but if you have just your standard analytics team also doing testing, you will never achieve anything close to the value that you can and should receive.

The limitations of this type of structure is a heavier need for sponsorship to allow the freedom and align the teams on central goals. You will now have personal or team goals for the various business units that may be opposite of the central or optimization team goals. It is extremely easy when resources are not “owned” for those to go to projects based on political or popular reasons and not the value they may bring to the organization. It is easy to talk about working together and having access to resources, but that tends to last only as long as there is not a fire that someone feels needs to be put out. Other limitations are the constants pull to do other types of work, especially for report pulling, as well as the need to make it clear what people are measured on despite being on very different teams. Despite those limitations, a lot of good can come if you have clear and strong leadership, accountability, and the right people in place. This is one of the most common structures for mature groups, and is one that will allow some level of success and expansion throughout the organization.

Framework #2:

In this second example, you see that you have a full optimization team, one without dotted lines and one who is independently part of the entire data team. Here you have technical and creative resources, but who only tangentially part of the larger marketing and IT teams. These resources are not directly part of their team, but the same group continuously work together to grow and expand the organizations testing efforts. The benefits of this structure is the ability to really develop the skills of the members, the central role within the larger organization, and the ability to have a separate charter and to really focus on improving the site as a whole, not just the smaller components within.

This is the preferred structures that I have seen for larger organizations. This allows for consistent resources, the ability to do the right thing for the site, not just the business unit, and an independent role for analytics and just marketing. It doesn’t create confusion for the team to do things that help only one group or person, and it allows for a clear line between testing and any other group. The goal is to work together to improve, not for teams to fight over who gets credit. The limitations are the need to constantly be working with and educating the various business units, and the complications of owning the impact for the team’s actions.

In order for either framework to function properly, the executive sponsor must take an active role in keeping people aligned and accountable for site goals, not just personal goals. You will never be able to function if you cant first deal with petty infighting and a lack of accountability towards a common central goal. If you do not have that, then don’t wait for it to happen and instead seek out sponsorship to make it happen. You don’t need or even want higher level executives to be dealing with the day to day operations, but you need an umbrella by which to separate the disciplines, align the resources, and make sure that the system itself is being used to its highest function. As much as we may want to pretend we live in a world where everyone works together towards a common goal, it is rare that this is the case. The executive sponsor’s primary responsibility is to raise the level of discourse away from petty individual goals and to hold people accountable for actions that make everyone better.

These are but two of the more common examples I have seen for successful programs. In all cases, the real success is far more about developing skills, challenging ideas, getting buy-in and accountability, and more than anything treating optimization as an independent function that never starts and never ends. The team is meant to be the best friend of all business units, not their worst enemy. It is meant to show them the efficiency of their efforts and to make sure that what they think matters really does. If you are just going to leverage testing to just push the same ideas, to prove value for other tools, or to make someone look good then you have no chance of building a world class organization. This means that you have to fight the battles to get past the initial resistance and educate people as to why their ideas won’t work and why you must challenge all ideas in order to understand the value, not just directly but relative to other courses of action.

There are no magic bullets to make your organization a great optimization organization. No matter what structure and path you choose to go down, it takes time and a lot of hard work to get people to understand how to make it work. The hardest task is always going to be getting past poor misconceptions and petty internal battles for control and who gets to claim success. Once you have gotten past those points, aligning in a way that insures success is the next step on the path to a great program. Programs fail when they stop fighting necessary fights and just go with the flow. If you want to make your organization the best it can be however, you can never stop improving and you can never stop trying to get people to align in the best ways possible.

Optimizing the Organization: The Mental Evolution of Programs

Building a true optimization program in your company can be a daunting experience. No matter how much you might want to make things work perfectly, the newness of the concepts presented, the politics around who is “right”, and a hundred other factors conspire against you. Most people speak about wanting to get good results, but are often unwilling or incapable of changing their own behaviors, let alone others, in order to get those results. Even worse, there are very few people who have actually built a world class program, and they are drowned out in a sea of “experts” who have the one thing you need to do to succeed. With all that information, where you are mentally about your program speaks volumes about the value you are getting and what the next steps are to really become world class.

No program is perfect day one, and almost all of them have to go through some very difficult growing pains before they are even functional. It is true that every program follows a similar pattern of evolution, but all programs risk eventually stopping their evolution due to a lack of will or understanding. The challenge starts with the mental evolution of the program, since the functional parts are mere reflections of where people view testing. It is important that you understand where you are, and where you need to get to in order to succeed.

The challenge is that all programs reach a stopping point, either through mental exhaustion, political pushback, personal ego, or a hundred other reasons. The key to becoming a top program is to get past that point and continue down the path, even when it seems daunting or does not seem to help you advance politically.

The mental evolution of programs:

Random Testing –

All groups start here, thinking of testing as a one off action you take to figure out which piece of creative to show, or which landing page is best. No program is able to achieve the efficiency that is necessary for their program when they are stuck at this phase, yet most conversations around testing and a great many programs never get past this point. One of the key reasons for this is the comfort and the easy to grasp nature of this stage. This is what most people think of when they think testing, and that is a shame since so many will never see the power it can truly bring to their organization.

The key signs of this stage are: “better” testing, each test is an individual project, you need to get approval for each test concept, you have no rules of action. Fundamentally you are focused on finding out who or what is “right”. Testing is a one off project that you do when you need to make a decision. Put succinctly, if you are talking about what you want to test, instead of letting results tell you what to test, then you know you have not moved past this phase.

One of the other major signs of this stage is the lack of aligning on site goals. If you have not gotten alignment on a single success metric for all tests, then I can guarantee you are at this point. If you are stuck thinking about specific metrics for a test, or think that you will decide how to act and what is important when you get the results, then you are firmly stuck in this phase. If you try to think or act on the data from tests the same way you do the data from analytics, you can not ever move past this point.

It is possible to get value at this stage, and the sad reality is that most groups never leave this stage, but ultimately you will never have a real optimization program and will be getting pennies on the dollar return if this is where you keep the conversation. Groups that are stuck in this way of thinking often think that more tests means more value, and that is true if you are leaving the outcome of a test to random chance. Groups that want to be efficient and to get real value from their program however need to apply those resources not towards running hundreds of tests, but instead towards shifting the mindset of the program to insure higher returns and more long term value from each and every action.

Long Term Site Integration –

The next evolution is to start tying in testing into larger projects. Working on a redesign? You start testing out smaller portions on the way. Focused on personalization? Then you start testing out different pieces of content or you start testing for different segments. Testing has shifted from being a random action of choosing between to choices to one that can shift and change the entire direction or path of a project. To reach this stage, you must be willing to shift some part of the project away from what you want to do, and instead choose to do the things that the data tells you to do.

The benefits of this stage are the start of organizational building blocks that are fundamental to a successful program. You will have to have agreed on what is success, you are starting to look at testing as an efficiency tool, and you will have built out some processes to make testing more efficient. Most likely you have some more dedicated resources and have testing as an ongoing thing, one that is not just a novelty one off. You are stopping testing what you want, and are letting some results from the test determine the path of a project or initiative.

The limitations are that you are just doing more of the same. You have not really built out a full program and you are still focused on “better” ideas. You are just creating more structure for the randomness of the previous phase, and it is starting to shape a direction, but you have not bought in to testing as a means to the question instead of just a way to find an answer. All groups have to get through this stage, and the ones stuck here are going to get more value from the random testing of the earlier phase, but it is important that you focus on moving to the next stage, which is the largest divergence on the list.

Disciplined Base Testing –

This is the real litmus test of programs and the largest gain and divergence point. Very few groups make this leap, but the ones that do see testing as a very different and more valuable component of their entire organization. The keys to this phase are a movement away from “better” testing and a change to open ended looks for the most efficient ways to apply resources. You are no longer looking to test out what you want, but instead using testing to understand the value of different alternatives and letting the results dictate the path of your tests and your initiatives. All tests do not predispose an outcome, instead focusing resources constantly towards the most efficient answer from the prior actions. Test ideas and appeasing CXOs becomes secondary to the discovery process and the opening up of testing to dictate its own path.

A sign that you have reached this stage is that you no longer look at just a test result, but instead focus on the value of outcomes relative to each other. You measure outcomes by the value of relative actions and not just that you went from 1 point to another. If you are not proving yourself and others wrong constantly, and if you are not humbled by the fact of how little you really know, then you have not reached this point.

The benefits are a constantly growing understanding of your site and users, and a move to ensure that all efforts are focused on the most influential sections and in the most efficient manner. You are no longer worrying about a “roadmap” or about what won, but about the process of figuring it out and constantly acting. You are starting to build out the trust that the system is only as good as the input, and to not worry about who is “right” as opposed to providing quality differentiated alternatives. You are learning with every action, and you are constantly stopping current paths that your organization is on and the causal data is discovering the value of new paths that you never thought of or would not have normally pursued.

The limitations of this stage is that you are going to upset a lot of people. You are going to be constantly proving that what people have held dear and believed as the core to their benefit to an organization is actually negative. You are going to show that myths passed down for years from schools, experts, and the very thing that CXO people hold dear is wrong. If you have not built out a culture where you are focusing on being wrong, where the goal is actual success and not propagating someones agenda, you will be dealing with constant headaches and internal strife. It takes a special type of person to stand up to pressure and to do what is right, even if it is not always in their best interest. If you are not willing to pursue results over “glory” then you will most likely not ever reach this phase in your programs growth.

If you have dealt with those issues, even if with just one or two groups, you will see dramatic improvements to the efficiency and return on your efforts.

Constant Iterative Testing –

There are very few organizations that have reached this point. At this phase, all parts of the site are open and constantly evolving using discipline based testing methods in order to grow and to get more efficient. There is no longer a view of a project at all, but a constant use of multiple resources to have a shifting and evolving user experience. All jobs in the organization have optimization as part of their duties, and no longer are you having debates about what you should test and what you think is better. Everyone is aligned on a common goal of growth and of proving each other wrong, not right. It is not about the test proving anything right, but instead about the quality of the input that is used to feed the system, which is starting to dictate just about every part of your user experience and internal resources and initiatives.

Getting a program to build out the mindset that allows for this phase usually means that you have dealt with all of the negatives that might arise, and have people aware of the benefit of being “wrong“. There is nothing that will show the inefficiencies of your organization faster then trying to constantly do things that might “hurt” someone. If you are not willing, capable to deal with the previous phases, most likely you will not come close to this level in your program.

Optimization Organization –

I refer to this as the mythical unicorn, as there are no organizations in the world that have reached this “nirvana”. At this point, all concepts feed the system and the system dictates the outcome. This is about letting go of worrying about who is right, and instead understanding that you still have to feed the system, but making the final decision has to be left to a disciplined (non-biased and predetermined) use of data, especially causal data, to make those decisions.

No group has reached this point because all organizations are run by people, all of which have their own agendas and all of which need to prove themselves “superior”. I don’t expect there to ever be a point where people are capable of putting their egos and politics at check, but working towards this point is the only way to really make a true difference and not just use data to push an agenda or to promote yourself.

The evolution of programs is a tricky thing, and not one that is quite as black and white as this path might make it seem. What is important is that you understand that you have to shift how you think, and be willing to change all the pieces that follow, in order to be successful. If you are still thinking about testing as a means of choosing between two items, or if you haven’t built out a culture where being wrong is more important then being right, then all the resources in the world wont make you successful. Success is not a random thing, and it is not dictated by how many resources you have, but instead by your willingness to prove yourself and others wrong. Building out the right mindset determines the long term value you receive, yet very few take the time to really understand or educate others. There are plenty of material out there that is happy to make you feel good about whatever stage you find yourself at, but if you really want to get value and really make a successful program, nothing will top constant hard work and the willingness to challenge the norm and to do things against “best practices”.

Understand the math behind it all: Normal Distribution

If the N-Armed bandit problem is the core struggle of each testing program, then normal distribution and the related central limit theorem is the windmill that groups use to justify their attempts to solve the N-Armed bandit problem. The central limit theorem is something that a lot of people have experience with from their high school and college days, but very few people appreciate where and how it fits into real world problems. It can be a great tool to act on data, but it can also be used blindly to make poor decisions in the name of being easily actionable. Being able to use a tool requires you to understand both its advantages and disadvantages, as without context you really achieve nothing. With that in mind, I want to present normal distribution as the next math subject that every tester should have a much better understanding of.

The first thing to understand about normal distribution is that it is only one type of distribution. Sometimes called a Gaussian distribution, the normal distribution is easy identifiable by its bell curve. Normal distribution comes into existence because of the central limit theorem, which states that any group, under sufficiently large number of independent random variables, and with a continuous variable outcomes, the mean will approximate a normal distribution. To put another way, if you take any population of people, and they are independent of each other, then an unbiased sample of them will eventually turn into an attractor distribution, so that you can measure a mean and a standard deviation. This gives you the familiar giant clumping of data points around the mean, and that as you move farther and farther away from that point, the data distribution becomes less and less in a very predictable way. It guarantees that an unbiased collection done over a long period of time, the mean will reach normal distribution, but in any biased or limited data set, you are unlikely to have the a perfectly normal distribution.

The reason that we love these distributions is that they are the easiest to understand and have very common easy to use assumptions built into them. Schools start with these because they allow an introduction into statistics and are easy to work with, but just because you are familiar with them does not mean the real world always follows this pattern. We know that over time, if we get collect enough data in an unbiased way, we will always reach this type of distribution. It allows us to infer a massive amount of information in a short period of time. We can look at distribution of people to calculate P-Score values, we can see where we are in a continuum, and we can easily allow us to group and attack larger populations. It allows us to present data and tackle it in a way with a variety of tools and an easy to understand structure, freeing us to the steps of using the data, not figuring out what tools are even available to us. Because of this schools spend an inordinate amount of time in classes presenting this problems to people, without informing them of the many real world situations where they are may not be as actionable.

The problem is when we force data into this distribution when it does not belong, so that we can make those assumptions and so we act with a single measure of “value” of the outcome. When you start trying to apply statistics to data, you must always keep in mind the quote from William Watt, “Do not put your faith in what statistics say until you have carefully considered what they do not say.”

There are a number of real world problems with trying to force real world data into a normal distribution, especially in any short period of time.

Here are just a quick sample of real world influences that can cause havoc when trying to apply the central limit theorem:

1) Data has to be representative – Just because you have a perfect distribution of data for Tuesday at 3am, it has little bearing on being representative of Friday afternoon.

2) Data collection is never unbiased, as you can not have a negative action in an online context. Equally you will have different propensities of action from each unique groups, and with an unequal collection of those groups to even things out.

3) We are also stuck with the data set that is constantly shifting and changing, from internal changes and external changes in time, so that as we gather more data, and as such take more time, the time we take to acquire that data means that the data from the earlier gathering period becomes less representative of current conditions.

4) We have great but not perfect data capturing methods. We use representations of representations of representations. No matter what data acquisition technology you use, there are always going to be mechanical issues which add noise on top of the population issues listed above. We need to focus on precision, not become caught in the accuracy trap.

5) We subconsciously bias our data, through a number of fallacies, which leads to conclusions that have little bearing on the real world.

In most real world situations, we more closely resemble multivariate distribution then normal distribution. What this leaves us with is very few cases in the real world that get the point that we can use normal distribution with complete faith, especially in any short period of time. Using it and its associated tools with blind loyalty can lead to groups making misguided assumptions about their own data, and lead to poor decision making. It is not “wrong” but it is also not “right”. It is simple another measure of the meaning of a specific outcome.

Even if the central limit theory worked perfectly in real world situations, you still have to deal with the differences between statistical significance and significance. Just because something is not due to noise, it does not mean that it answers the real question at hand. There is no magical solutions to remove the need for an understanding of the discipline of testing nor the design of tests that answer questions instead of just pick the better of two options.

So how then can we use this data?

The best answer is to understand that there is no “perfect” tool to make decisions. You are always going to need multiple measures, and some human evaluation to improve the accuracy of a decision. A simple look at the graph and having good rules around when you look at or leverage statistical measures can dramatically improve their value. Not just running a test because you can, and instead focusing on understanding the relative value of actions is going to insure you get the value you desire. Statistics is not evil, but you can not just have blind faith. Each situation and each data set represents its own challenge, so the best thing you can do is focus on the key disciplines of making good decisions. These tools help inform you, but are not meant to replace discipline and your ability to interpret the data and the context for the data.