Because of the nature of how most organizations work, it is common to find testing added onto the existing roles or organizational structure of the analytics group. It makes sense from a very high level view as both disciplines deal with using data to make decisions and both can be viewed as a shared resource throughout the entire organization and for each group. The failure comes however when people who are used to looking at and responding to problems like they are analytics issues try to force the same actions onto testing.

The real key to success in adding optimization to your organization is in how you tackle the fact that it is a new discipline then in the whom or where it fits into that larger picture. Most groups worry about head count and resource and fail to focus on the skills and different actions that determine success with optimization as opposed to general analytics or business intelligence work. The lack of knowledge by others is often abused or leveraged to help people gain oversight onto this project. The real problems lie however when people do not then adapt usage to meet the new needs and instead simply try and come up with stories as to the value of the program.

Once mature and with key people in place combining the programs can add a lot of value. Without deep understanding of those differences however, the inevitable conclusion is less value and wasted time as people make mistakes that they do not even know are mistakes. Resources are a precious commodity, and the ultimate expression of optimization is to leverage them in the most productive way possible. In order to ensure this outcome, it is important that you focus the time of your optimization team towards actions that will maximize outcomes and help grow understanding for your entire organization.

To maximize success and to ensure that focus is done on the right actions, here is a breakdown of the time spent and the value derived from the main actions of a successful optimization team.

1) Active Data Acquisition – 80% – 90% of the value of most optimization programs comes not from the commonly thought of validation role but in continual active data acquisition and comparison of feasible alternatives. It takes time for groups to achieve this role, but when they do the value given to the organization increased by magnitudes. Often this is based around the concepts of bandit based optimization and fragility and is used as an ongoing effort to challenge assumptions and to actively measure the value of different alternatives.

In this role the optimization team consistently leverages low resource efforts that consistently measure as many different feasible alternatives as possible and do this across the site. This is in the attempt to maximize the discovery of exploitable opportunities and the primary role is to challenge assumptions which otherwise will never be discovered. The team needs access to pages and clear rules on measuring success and leveraging of resources in order to produce constant and impactful lessons which can shape and direct product discussions and roadmaps.

2) Education – This is 5-15% of the value derived from the optimization group, but provides the ability to do the actions which produce the greatest returns. Because optimization requires very different ways of thinking about and executing on actions in order to provide the most value this means that one of the key roles for optimization is an ongoing and consistent conversation with groups about different ways to think about problems.

It is vital that this conversation always happens prior to any action and is that optimization is not just thought of as a simple action in a release calendar. Groups that fail to think different are guaranteed to get much lower value from their testing efforts, waste far more resources, and often have much slower and less productive product teams overall. They will get results, they will simply be far smaller results at a much higher cost. Failure to focus on education often leads group to a purely responsive role and leads to programs that are happy with the number of tests they run or by simply producing a single positive result.

There is no such thing as an organization that starts out looking at the correct things perfectly and without fail someone’s personal agenda leads them to subconsciously search out confirming actions and data in order to make themselves look good. Building, maintaining, and educating people on proper data discipline is the single most consistent and important topic of education.

Here are a few of the key topics that groups can and should focus on:

i. Rules of Action – Knowing how to act on data and to be disciplined in not acting too fast or too slow and looking at only metrics that matter to a decision are vital for any data organization.

ii. Statistical disciplines – There are many different ways to think about data and testing and it is vital that people be exposed and open to different ways then they are previously aware in order to maximize future growth.

iii. Psychological disciplines – Optimization hits on many psychological disciplines and concepts such as Confirmation bias, Forer effect, Congruence bias and many others.

iv. Knowledge Share – When you are running a successful optimization program you will be constantly learning things that go against all previously held beliefs and opinions. These lessons learned are the single most valuable part of a successful program and become a core component of a program once it has matured.

3) Ad hoc analysis and validation testing At most this represents 5% of possible value provided by testing – It is always fun to want to focus on who has the best idea to improve things, but ultimately this is the least important part of a successful program. The better the input, the better the output, but only if it is going through a great system. A poor system means that it really doesn’t matter how good any idea is.

This is the part that most groups are familiar with, where they respond to test ideas directly or asking for more data / details on specific tests.

Generally this time is best spent redirecting towards higher value uses of time and value of data.

Successful programs have their time breakdown in the range of:

• Active Data Acquisition / Ongoing optimization – 60-70% of time
• Education – 15-20% of time
• Post hoc analysis and validation optimization – 15-20% of time

It is generally more about the thought and usage of the program then just who owns it. The real keys to a successful program are to differentiate the roles and skills. Generally if a program is just starting out you they may have 1-2 people who work on testing in some form, with the primary focus on working with different groups to work with their ideas to provide value. Mature programs might move upwards to 5-10 people and higher as they continue to grow.

It does no good to add more people to a problem if you aren’t fixing the real problem, which is how the time is spent. More time does not equal more value, better usage of time means more value. It is never easy to go past your comfort zone, but that is where you will find all the value. Think about how and where you are spending your time.