Rant – Statistics is a Tool & Not an Answer

I read a post by a famous blogger today that brought up one of least favorite things about our industry. One of the funniest things in our industry is the over reliance on statistical measures to prove that someone is “right”. Whether it is Z-Score, T-Test, Chi-Squared or other measures, and people love to throw them out and use them as the end-all be-all of confirmation that they and they alone are correct. Besides the hubris of the situation, it is just a new tool for people to abuse and not understand.

What is funny about this is they are some of the worst true measures of impact and importance of data (or “who is correct”). They work great in a controlled setting, and with infinitum data, but in the real world, they are just one of many imperfect standards for measuring the impact of data and changes. You cannot replace understanding patterns,looking at the data, and understanding its meaning. I would settle for people understanding what those tests even tell you (hint, it is not 95% confident you will get a 10%lift).

It is like someone found this new fancy tool, and suddenly has to apply it because they realize that what they were previously doing was wrong, but now, this one thing, will suddenly make them perfect.

For the record, three really simple steps to measure impact of changes:

1) Look at performance over time – Look at the graph, look for consistency of data, and look for lack of inflection points(comparative analysis). Make sure you have at least 1 week of CONSISTENT data (that is not 1 week of data). This also gets into why visitor based metric systems are much better for this analysis, and also why you need to think in terms of propensity of action.

2) Make sure you have enough data – The amount needed changes by site. Some sites, 1000 conversions per recipe is not enough, some sites 100 per recipe are. Understand your site and your data flow. I cannot stress enough that data without context is not valuable.

“Information is not knowledge”

3) Make sure you have meaningful differentiation –Make sure you know what your natural variance is for your site (in a visitor based metric system, it is pretty regularly right at 2% after a week). Make sure that the lift is consistent, and that it is more than mechanical noise (something laboratory stats equations DON’T account for). You can be 99% confident at .5% lift, and I will tell you have nothing (neutral). You can have 3% lift and 80% confidence, if it is over a consistent week (this is not daily performance) and I will tell you have a decent win.

I am not saying that there is not a lot of value from P-value based calculations, but I will stress that they are not panacea nor are they an excuse for not doing active work to understand and act on your data.

Rant – Starting Out

There are a number of newer consultants and newer people in our industry that I have been working with lately, and that has led to a very common question:

“Starting out, what is the number one thing you suggest that I focus on learning?”

I don’t know if I can give one thing, but I can definitely give a few things:

1) Understand how businesses move and make decisions – Business DO NOT MOVE RATIONALLY, nor are they easy to move. There is an art to changing the direction and most importantly, knowing how to say NO to someone and get them to move to the path they need to.

2) Understand that interacting with clients is the job, and that having a strategy for the engagement, and then acting proactively on that strategy, is the name of the game. Don’t ever get stuck being reactive.

3) Start learning the disciplines of optimization. Don’t ever focus on the tactical actions, but instead focus on the disciplines, why you do something, and why you don’t tackle problems like other disciplines

4) Start understanding cognitive biases. You will deal with them all day long, and you better be able to understand and cope with your clients and your own.

What the job most definitely is not:

1) Learn how to do everything the tool can do… You will learn the tool, but information without context is not knowledge. Any monkey can learn a tool, but it only has as much value as how it is leveraged.