Most marketing people have only a passing interaction with statistics, and often times only understand it as a measure of how it has impacted their daily life. One of the funny things people don’t realize is that there are two completely different competing schools of thought when it comes to statistics. Most people are familiar with frequentist statistics, having dealt with things like normal distribution, bell curves, and established probabilities. The other school, Bayesian statistics, is a realm that fewer people are familiar with, but just as applicable. In fact, the move over the last few years is for more people to change from the frequentist model to Bayesian techniques.
So what is Bayesian statistics? To put simply, Bayesian analysis is the use of conditional or evidential probabilities. It looks at what you know of the environment and past knowledge, and allows you to infer probabilities based off of that data. It asks what is the likelihood of something happening based on our knowledge of past conditions and the context of them in the world. Where frequintist statistics can be viewed as much more of a evaluation of the larger data collection and judging the chances of something happening again based off of those results, Baysian is about the likelihood a set of results reflects the larger reality and about making inference based on the limited data set.
Whereas a frequentist model looks at an absolute basis for chances, something like the population of females is 52%, so that means that if I select someone at random from my office, I have a 52% chance of picking a female. The chances are purely based on the total probability. The Bayesian approach is to rely on past knowledge and then adjust accordingly. If I know that 75% of my office is male, and I grab a person, then I know that I have a 25% chance of picking a female.
So is it 52% or 25%? Both are correct answers depending on what question you are really asking, but both look at things differently. Frequentists look at the larger perspective of all chances, and base things off that ideal look at the world. Bayesian users use much more personal or past knowledge to infer information. Bayesian thinkers would much rather answer what are the chances that the total population is 52% female based on the fact that only 25% are female in this office. The risk with using Bayesian logic is that you are allowing for bias and poor data collection to dramatically later how you view things. The gain is that while frequentist will often be right in a controlled setting and over time, Bayesian has the chance to give you better information based on what you know. Bayesian logic also allows you to do conditional logic statements, like based on the office scenario before and a little bit more contextual knowledge, you can answer “what is the likelihood that if you choose a women that she would have blond hair?”. Bayesian techniques are often used for logic reasons, because it allows you to make a conclusion about the likelihood something is the best answer based on what you know. Both techniques are at risk for black swan type of analysis, though Bayesian analysis can be even more influenced by only focusing on the known.
So why is this important? All testing tools and models are almost always relying on frequentist techniques to give you the global view of something as to how often it fits into a pattern. This is why you see things like 92% confidence when evaluating things, we know that under similar circumstances, 92% of the sample means will fit into that window. Those techniques give you answers in an ideal situation and over time, but that may not be true of specific periods or non normal events. They don’t take into account the context of this specific situation, nor prior history relevant specifically to the situation. They often times don’t take into account even the contextual knowledge of the other recipes and information contained in that same test. They might be true of normal circumstances, but not of a special sale or seasonal activity. Bayesian techniques rely on prior knowledge that for testing is rarely available, and for analytics is problematic at best. They might reflect special circumstances, but not give a good long term view due to those same mitigating circumstances.
In all cases, nothing will replace understanding the context of what your data tells you, the patterns of it, and knowing how and when to act. You have to appreciate what the statistics are telling you, but also appreciate what they aren’t telling you. Any overt belief in a measure, by itself is always going to be problematic. Just getting a statistical answer is not a replacement for the context and the environment by which you are gathering data, nor making a decision.
No matter what techniques you use, no matter which camp you are in for the correct way to look at things, there is never a time when you can ignore the problems of any single type of analysis. You can not replace using discipline and logic in your actions. Statistics are just a tool, they can not replace proper reasoning, yet too many people look at it as a magical panacea to remove responsibility for action. Always remember that there are multiple ways to look at a problem, let alone hundreds of ways to solve it. Figuring out the efficient and best way for you is the real key.