As more and more programs start to emerge with the growth of the online optimization field, there becomes a preponderance of “best practices” when it comes to testing, personalization, and all other active forms of leveraging data, it seems like you have to know a massive amount to just understand completely what those “experts” are saying.

With that in mind, I wanted to present some very simple do’s and don’ts for programs just getting going. Starting correctly and setting the stage for success is vital to really being efficient and getting value out of your program that you can and should. The problem is that in almost all cases people’s first instincts lead them astray. What you don’t do is more important usually then what you do choose to do. The key is to make sure that you focus your limited time on the actions that will provide the greatest growth and value to your program. The same advice can work for groups that have been testing for years as many of those programs also are just built up versions of the same bad behaviors.

DO – Hold discussions about a single success metric

The very first and sometimes the most painful hurdle that a program faces is getting groups to agree on how to act. This is in many ways completely counter culture as many groups have competing goals and are only focused on their little piece of the larger pie. If you do nothing else, getting people to agree on the one thing that you can all make a decision on is vital.

A side benefit of this conversation is that it starts the process of allowing people to dissociate the actions they think will lead to success and the actual measure of success. Way too many people think that if their idea is more people looking at product X will generate additional revenue then the measure of success is more people looking at product X. You may have an idea for what you want to do, but you are doing it to accomplish a goal, so measure the goal, not the action. The measure of success would be additional revenue, and once that is the only goal, you can start comparing all feasible ways to achieve that exact goal.

DON’T – Get too caught up on test ideas

Some of the least important parts of a testing program is the generation of test ideas. While this is the fun part for people to try and prove their point, the keys to success are not in having a bunch of ideas, but in putting together the infrastructure and helping understand the discipline of successful testing. Test ideas will come naturally out of everyday conversations and especially out of prior tests and learned knowledge. There is never a lack of things you can do, but focusing too much on that part allows people to get caught up in many different biases which will make their rational evaluation of the results to collide with their ego.

DO – Apply tech resources on a larger infrastructure

All tools require some sort of deployment, and while some are easier than others, the biggest mistake you can do is to think that every test will require massive amount of resources. If you build a proper infrastructure across your site, then most tests will not require any involvement from development resources whatsoever.

The key to a good infrastructure is to have tagging in the key locations on your top pages so that you can test just about anything. You will also need to make sure that you have tracking in place for your success metric, and for any additional information (like segment information) that you may want to provide.

Testing should not be thought of as a project but as an ongoing organization and site feature. It is something that should be set-up in a way to never stop and to never be about the simple validation of a single idea. In order to maximize this, going through the initial “pain” of a larger deployment and making sure that your IT group understands that this is not a permanent engineering owned project will dramatically improve your ability to move quickly later on. The key once this is done is to prioritize tests based on resource usage and prioritize tests that will deliver the greatest return for the lowest resource usage.

DON’T – Think testing is just an extension of your analytics group

How you think about optimization is almost the exact opposite of analytics. Instead of patterns and anomalies of larger data sets, you have a single point and the push to make consistent meaningful changes. Testing is not just the action arm of some analysis you did to validate your point, it is the active acquisition and interaction with data.

To succeed, you need to think about segmentation differently. You need to think about what a success metric really is and how it is different in testing. You need to able to speak in terms of comparative analysis, not validation. Basically, you have to be able to turn just about everything you do with analytics on its side. Later on, you can start leveraging the two together, but as you start, separating them completely is going to grant you far more return with far less work then trying to just tack testing into your analytics daily activities.

DO – Think about your rights management

Make sure you know who is going to have what rights and make sure you have some checks in place from too many people changing your site.

DON’T – Blindly follow statistical measures

You don’t need to know everything about all statistics, but you do need to understand some basic concepts to really understand results. The first is that for any statistical tool to be useful, you need not just statistical confidence, but you also need the data to be representative of the change you are going to make. If you get 99% confidence in 3 hours on a Friday afternoon, that data is only representative of that period of Friday afternoon.

DO – Starting thinking how you are going to store and share results

When you are testing right, you are going to constantly learn new things and if you are doing your testing right these lessons you learn will eventually be far more valuable than any individual result. You need to start thinking about where you are going to share this information, the format, and the availability. You also need to make sure that this is not a static item but a living knowledge base.

DON’T – Let any test go out with just two recipes

One of the hardest lessons to learn is that testing is not about validation of a single point, but about comparing feasible alternatives and being prepared to go in directions that you never imagined. While not everyone will be ready for this day one, the simplest way to prepare people is to force discipline on them. Making people have multiple very different but feasible alternatives will start giving you far more information and will start to show you areas where what they thought mattered didn’t.

There are a thousand other things that go into running a program, but just starting out, if you tackle these simple things and avoid some common traps, and then you will be get far more results, make a larger impact to your business, and use far less resources. Think about what you really want from your program and then stop focusing on individual tasks and instead start putting the key pieces in place for long term success.