As I see one again one of the surest signs that you don’t know what you are doing in testing, I am forced to clarify one of the greatest misconceptions when it comes to testing.

The first rule of any statistical analysis is that the data must be representative. It doesn’t matter how statistically accurate your count of blue cars is if you are trying to measure the impact to the entire freeway…

So why then do people think tests should only be to “new” users. This is one of the most consistent misunderstandings and confusions of cause and effect. People who have been to your site have come there for a reason and are coming back for a reason. Just because they saw the old site, it does not mean they are not fundamentally important to your new analysis. They are not repeat users just because of the new site (causation), they are people who have declared an intent and are researching or repeat interactiors with your brand and products. They represent anyone who has previously wanted or been interested in what you are selling who have happened to have been on your site before (correlation).

This means that while there is some possible interaction with seeing an old and new experience, that ignoring them is you telling me that you do not care and are not interested in the revenue generated by anyone who has ever come to your site or purchased from you before or who has thought about purchasing from you before. There will be some interaction from the change in experience (if they even remember) that is spread evenly over each sample, but that is especially mitigated in a visitor based analysis (to measure performance over time). What is not accounted for when you do not allow them into your test however is 100% of all people who have given you revenue before!

That means that if you have any business where you would like people to repeat use or purchase from you, that not including them invalidates all your data, since your data set is both biased (people who have never been interested in you before) and not representative of your long term population.

To put another way to match this time of year, this is exactly the same of only polling fox news watchers to represent all of America for the presidential election. Or MSNBC watchers to vote for all social bills. It is a population, you will get a result, and if you really want to ignore reality, you can go with the data, but it in no way represents the entire population or in any way tells you what matters to the entire population.