Testing 202 – 5 disciplines to get greater value from your testing program

Testing seems like such a natural extension of most existing operations that very few groups take the time to evaluate as a separate discipline. They then are shocked when they hit the inevitable points of resistance that all programs run into. They are not prepared for the political, technical, or organizational barriers that riddle their journey. Even worse, those barriers interfere with the need for simple speedy execution, which hinders their ability to reach the level of monetary impact that the program is truly are capable of. What differentiates programs in the long haul are those that leverage their ability to push past those barriers and change the way they think about the testing, evolving beyond just focusing on simple tests. They start to focus on the power of the system itself to shape their forward trajectory. It takes people to be real thought leaders, not just for themselves, but also for their organizations, to really reach the next level of their program.

The theme of the first group of disciplines for programs was all about organizational consistency and getting everyone to work efficiently and quickly towards one goal. Once you have achieved that ability to move forward and act as needed, the main ingredient for success comes from your ability to think about testing differently, to challenge yourself and others to focus on learning and on not making assumptions. Way too many groups end up leveraging their testing program to only look a the impact of what people want to do, instead of using as a tool to learn and go in new directions. Often times the least efficient part of any system are the people running it, and as such we have to come up with ways to break down that barrier and to allow the test data to really dictate where we go, what we talk about, and how we allocate resources. How you think about testing, how you challenge yourself to not fall into the many biases that dictate human nature, and how you can bring people to challenge their own actions dictates the scale of the impact of each test and of the program as a whole.

So many groups run into the problem of not realizing that they are sub optimizing. They run a test, and they get a winner, and so they assume that everything is golden. They report the winner, they move on. We get so caught up in the immediate return on our actions that we never take the time to understand what it means in context. The problem is that we are not treating each action with the respect it deserves, and that they are taking they are busy looking at today and not tomorrow. It is never about getting a result, it is about the ability to differentiate different results from each other, to make sure that we are going down the most efficient path for our business. If you get a 3% lift, that is more then you had previously, but what if the 3% is just one recipe in a test with a 5%, a 7% and a 10% winner. Would you then think pushing the 3% winner was the best course of action? The only way to escape that trap is by changing how you think about testing, and the practices that you put in place around that change, you are given the ability to measure the efficiency of your actions, to view the causal relationships of alternatives and to measure the scale of impact and the cost to achieve them.

In order to make sure you are getting the most from the actions you have enabled, here are 5 disciplines which will help you achieve the results you want from the program.

Best instead of Better Testing –

I have already explored this concept here, but it is important to understand the distinction. “Better” testing is the act of trying to figure out if one idea is better than another. It limits the playing field and is used to make an immediate decision about who is “right”. Best testing is trying to figure out what the value of each feasible option is, and to figure out what the best places are to put resources or to move the site. Testing is a system, one that only produces value based on the quality of the input. If you limit your input to only popular opinion or a few ideas, then you are dramatically hindering the output of that system. It is about using testing to not just push preconceive notions but to instead democratize ideas, so that the system is more important than the idea. If you are able to let testing tell you where to go, you will not only get better results from this test, but you will better inform future tests and stop yourself from spending resources in an inefficient manner.

Best testing is the first step to allowing your program to produce exponential growth in a way that just a test would never be able to to do. You are able to use the program to build, not just run tests. The entire goal is to increase efficiency and to facilitate learning, and the easiest greatest step towards that goal is forcing your team to think in terms of what is best, not just what idea is better.

Focus on Learning –

Every test you run is the chance to learn about your site and to get outside of your comfort zone. So many groups fail because they only test what they think will win, or what they focus on trying to get a consensus about recipes. If you spend the time and energy you waste talking about test ideas and focus it on creating all the options, you will spend less time and energy and will get better results. Stop arguing and move those resources towards creating. If you are really trying to open up your testing to feasible alternatives, you will constantly find winners that fly in the face of crowds. If you are focused on that outcome, you suddenly find all sorts of new lessons waiting for you, with the added benefit of getting magnitudes of value on top of what you learn.

There are so many assumptions, misconceptions, and faulty “best practices” that dictate the online world. Even worse, we make assumptions that something that works elsewhere works for your site and your users. We gain nothing if we are just proving ourselves right, but instead when we challenge those ideas, we start to learn about what makes your site unique and what works best for what you do. We start learning about the best places to efficiently change your site, and even who the most exploitable user segments are. Even better, those lessons seep into your other conversations, to inform product plans and senior management and all other groups about what you know, not just what you think or pretend you know.

The most obvious example of this is with multivariate testing. So many groups, especially agencies, push MVT testing as a tool to find a single answer by throwing a number of variants for multiple items on a page. It is a big mixing machine to reach a new version of the page faster. If you change that and use MVT testing as a learning tool, to focus on what section of the page or what factor of a section is most influential, then you are able to leverage your resources in a way to maximize the ROI and learn, accomplishing both tasks far better then just throwing things up to see what sticks. Anytime you are running a massive MVT, full factorial or not, you are sub optimizing, both in a resource and time perspective, but also because you have failed to leverage the opportunity to learn.

Living Knowledge Base –

As programs grow, and as you use testing as a vehicle to learn, the most important thing you will accumulate is not lift, but functional knowledge about your site and users. Storing and sharing this knowledge, based off of causal data, informs future decisions in a way that analytics or just “Best Practices” will never be able to do. You have to make this storing and sharing a function of the team, and make it accessible and meaningful to all groups, even those that were not part of the test that gained the knowledge in the first place. For most groups, this accumulation and sharing of knowledge has scales of impact far greater then just the individual test results.

Building a repository of that knowledge, and having it be an active breathing thing, that interacts with people and exists outside of individual tests is vital to achieving the results that programs want to achieve. What it is not is just a list of tests run and their results. What it is meant to be is something that shares lessons, successes, and failure, but is focused on the learning you have done from your program, not the minutia of the actions that got you there. Every version of this is different, as is each and every organization, yet they share the same characteristics: they focus on what has been learned across tests, they are easily accessible, they are used to start conversations and as a barometer to go, and they see what does and more importantly what does not work. It also allows you to weigh the various actions against each other, as just lift alone does not tell you the scale of impact. If you continue to try the same things that fail, you will never be able to leverage the exponential efficiency that learning should allow you.

Iterative Testing –

Iterative testing may seem like a no brainer, but so many groups fail to understand or leverage it as a discipline, instead talking about but failing to act on it consistently. It should be an organizational rule that no test is ever “over”, but only at a new stage ready for the next test. If you are using tests to learn, then you will know how to prioritize a page, which then needs to have the different sections explored, for not only what the feasible options are, but also what the most influential parts of those are and then what the best way to tackle that winning factor may be. The goal here is to make sure that each test that you run maximizes the efficiency while mitigating the opportunity cost, and the only way to do that is to constantly build off prior knowledge to insure that you are maximizing your placement of resources. In many cases, this can be the most difficult barrier for groups to actually overcome, as while there is a lot of positive talk around the subject, there are so many pushes and pulls for your time and for the resources, that it is easy to lose track or to just stop at the end of any given test. You have to force yourself and your group to maintain that momentum, and more importantly leverage all of that learning, to really drive where you are and where you are going.

Here is an example:

You take a product page, you challenge yourself to learn about what the most influential section are, so you test out what belongs and what doesn’t by removing all possible sections on the page. You learn that 2 sections don’t matter, the top navigation and the brand information. You then test out removing them together and find that you have improved the page towards your site wide single success metric. At this point you have a new page after pushing the winner, but need to dive deeper. You then use a small 3X2 MVT to learn that the button is the most influential element on the new page. You follow that up to look at the factors of that button to figure out what about it drives that influence. You learn that color is far more influential that copy or size so you then test out 5-6 different colors, and learn that purple, the color that everyone thought never had a chance of winning, actually does better than all the other colors.

What you have is an entire path that you never could have preconceived. You have forced a way to make sure that popular opinion did not drive what you test. The resulting page is not one that anyone could have predicted, but is by far the best performing. You have learned what the importance of elements are, the best way to change them, and you have used very little in the way of resources. You are free to continue, as you optimize the second most important part of the page, or you optimize the second factor of the button. The process continues, either in that same part of the user experience, or where you can see other opportunities to do the same process based on what you have learned.

Iterative testing is something that is constantly thrown around, agreed on, but then why is it so rarely done consistently? Why do so many groups think that talking about iterative action is enough? Groups get too caught up on the single winner that they miss that it is just one part of a very fluid user experience. You have to force yourself to follow this path, that is why it is a discipline, and when things become difficult to push through and do this, always, in order to reap the rewards you are seeking. Even better, if you are using segmentation at each point, you will end up with a page that can be dynamic based on the winning alternatives for exploitable segments. Each “test” is really just the next evolution of the same process. You have to stop yourself and your organization from viewing the test as the ends to itself. Doing this consistently, not once but always, really differentiates the value you will receive from testing.

Deconstruction –

The ability to have someone present you a test idea, and then break it apart to find the assumptions that lead to it so that you can learn as you grow is a vital skill for optimization. One of the most common mistakes that testing groups make is to take each idea at face value. Every idea comes from only one point of view, and is riddled with biases. You have to force yourself and others to get past those points to really discover the value hidden behind what sounds like a conventionally accepted concept. Treat the most important part of your program as the system by which you discover new things or challenge biases, and you will always be able to get greater results. We need to take any idea, and challenge every core part of it, so that we leave nothing to chance and so that we can really evaluate what works, not just what sounds like it works.

So rarely are you presented with the chance to optimize a page from start to finish as mentioned above, but that does not mean you are not still responsible for applying the same discipline to any starting point. People will always come to you with test ideas, either through internal debates, feedback, or just as they are going across the site. There is most likely value from each idea, but the real skill is to break down the idea, to make sure you are not presuming the path, and to learn. Is that item even needed? Does what you have been doing, is it positive or negative? Is that the best place to put resources? What else can be done with that space? Being able to take an idea and challenge the components of it is how you will arrive at the most important lessons you will learn.

The idea of targeting content in your carousel on your homepage based on what people have already purchased sounds like a great idea, but lets evaluate all of those assumptions:

Is purchaser even exploitable (can you change their behavior by changing the user experience)?
Is it the most exploitable way to look at the same user?
Is the homepage the right place?
Is the carousel the most influential part of that page?
What type of changes to the content are most influential, is it the wording, the presentation, the location?
Does content have the largest impact?
How does that entire path compare to other alternatives?

That is just the very first pass. It is actually far more likely that just changing the layout of your homepage based on browser is going to be both much higher in yield, but also more efficient of a much larger scale.

It takes practice and discipline to force yourself to challenge all of these ideas. It can often lead to discomfort at first, but pushing past that and assisting others in seeing their own biases and their own assumptions helps everyone grow. Testing is not about who had the best idea, it is about becoming humble and realizing everyone is “wrong” and about creating a system to push past that and learn and challenge common convention for the betterment of everyone.

Conclusion –

I am often asked how I define a “successful” program. When it comes to measuring a program, I measure it on how often they have learned something new and unexpected. Have they done something that makes everyone go “That can’t be right”, or “that goes against everything I have ever heard.” The only way to have those moments and to get the magnitudes of value both short and long term that those conclusions present you is to break apart ideas and to challenge yourself and others on the questions that you are trying to tackle. It isn’t about being sadistic or altruistic, but instead about looking at the discipline as a means to an end that helps everyone achieve their goals.

In the end, it doesn’t matter if you can run 100 tests a month if you are running sub optimal tests. The point is never what you did get, but what you got in relation to what you could have gotten with the same or fewer resources. So many groups get lost because they focus on the actions, and not the disciplines that define them and the larger picture of the program that they exist in. You did an action, that you might have done anyways, but it is over and you are left with nothing but the next idea.

You have to change how you think and how you act to get the results you want and to make testing a fundamental part of who you are as an organization. I talk in terms of disciplines, because that is what they are, they are core beliefs that only take hold when we challenge ourselves to live by them every day, not just when it is convenient or easy. Challenging yourself to these disciplines and asking the tough questions of others will allow you to move down a path away from the obvious to the world where you learn, grow, and have results that really move people.

To navigate the entire testing series:
Testing 101 / Testing 202 / Testing 303 – Part 1 / Testing 303 – Part 2

Testing 101 – 5 disciplines to grow your testing program

The beauty of any program or any skill is that no matter where you are, or what you have accomplished, you can always get better. There is no such thing as the perfect testing organization, or a perfect way to use data, which means there are always opportunities to grow and to get better. With that in mind, I want to present various skills and disciplines that will allow your program to grow no matter where you are at. If you have just started out, been testing for years, or are a massive optimization organization, there are always new ways to think and new skills to allow you to get more value out of each and every action that you take.

All programs are built off of key fundamental disciplines that allow all other disciplines to grow and to be acted on accordingly. All groups start out by needing to grow past the initial challenges and limitations that face your program. Most groups find a point where testing has shown a few wins, you find places to test where it makes sense, but you haven’t been able to get the level of consistency or buying that you are looking for. You are trapped in a world of explaining why you want to run a test, or working with different groups who aren’t interested in what the rest of the organization is doing. Early development programs have many factors that unite them; you might have been testing for 3 days or 3 years, and yet if you haven’t really taken the time to grow and incorporate testing beyond a superficial way, you will never achieve the true value that testing can provide for your organization.

The first thing that differentiates programs is the organizational pieces that have been put in place to help them move at a speed that will truly impact the bottom line. With that in mind, here are 5 key disciplines that will allow your program to the next level. For any testing program to work long term and to grow, you must first address the organizational roadblocks that will without fail trip up your program. The first and sometimes most vital aspect of growing your program can be adoption and the realization that this is not a once in a while type of action that you take.

Single Success Metric –

Without fail, the single greatest determining factor in the overall success of your testing program is have you unified on what you are trying to achieve and are you optimizing for the end behavior you want. So many programs fail when they try to optimize for the next page or a small action, or even worse to try and optimize for multiple metrics to make everyone happy. So many programs are less the sum of their parts, and because of this they are never able to impact the fundamental bottom line of the business at the level that they are capable of.

It isn’t just about optimizing for the goal of the site and not focusing on a individual concept, you must be able to answer all questions of this format: If A goes up, and B goes down, what do we do? What about A and C? Ultimately, you will quickly discover there is one thing that you really have to go up to be successful. It isn’t about clicks, our bounce rate, or just moving people to another part of your site, it is about focusing on what all of those behaviors are trying to accomplish. In just about all cases, the metric you will end up using is directly tied to how you make money, something like RPV, or lead conversion rate, or an engagement metric. What is important is that is global across your entire site, not just the section you think you will impact. That is what you need to focus on. You need to have everyone in agreement on how to act on the test before it launches, so that you don’t lose momentum or get lost in the internal quicksand of politics.

Nothing allows your program to be more valuable and for you to bring people together then to have everyone pointed towards the same end goal. Nothing ends internal debates faster than not worrying about clicks or sign-ups or search versus organic then optimizing for revenue and giving all actions an equal footing.

When I come in to assist groups and make sure they are getting the most value, nothing is more vital and I will refuse to go forward unless we can get agreement on what defines success and how we are going to measure it. You are often defined by what you don’t do as much as what you choose to do, and this is the seminal moment when you can take a giant leap in your program. This is the moment when a new discipline, unique to optimization, comes into place. It also allows us to treat any idea democratically and let the numbers define success.

Consistent Resources –

One of the many failings of early programs it he lack of ownership of the process and tool. Does IT, marketing, product, merchandising own testing? Whose resources get assigned and who will run this specific test? Do you have a FTE, or just a few random hours assigned to someone who has 10 other tasks on their plate? It is vital that as you build your program, that you have consistent time and resources assigned, outside of any specific test or project. The skills of testing, not just the tool, but understanding how your site works, how to move things, testing disciplines and how to navigate internal politics, all of these take experience and time. Programs that have new people jumping on board, people who run a test once or twice a quarter, or who have not clearly identified who will set-up, run, and do reporting on tests have the hardest time creating the momentum and cadence that allow your program to really blossom and to help insure that you are getting the magnitudes of value that testing can provide.

A good first step for any program is to assign a set number of hours per week. Assume that you will always be testing, and such, this won’t be tied to any individual test. As a good next step, you can make sure that any test you run can be done with that amount of resources in one week. Assigning the same people, a set amount of time, and making sure that you use it to always be moving forward will help your program achieve the cadence and scope of impact that you are looking for.

Infrastructure –

Having a proper infrastructure on your site is vital to be able to have a real program. The pieces of your infrastructure has two main components, the site code and the organizational components. A good robust infrastructure on your site should allow you to get a new test live within a day on any important part of your site without IT involvement. If you are creating testing code for each test, or if you require a separate ticket and full QA process for each test, you will never be able to move at the speed that a successful program runs at.

The other component for infrastructure is the organizational components. Do the important groups in your organization understand how you will run a test? Why you run a test? What will define success and how best to think about testing to provide meaningful answers? Are they bought in to testing or do they see it as something that threatens their imminent domain? Whereas the technical infrastructure is something that may be done mostly at the start and then revisited from time to time, the educating and working with various groups in your company is an ongoing thing and is vital for long term growth.

Rules of Action –

Once you have figured out your single success metric, do you have a consistent set of rules to define how you call a winner? How about when you have a winner, how does it go live, who is responsible, what is the timeframe? How do you build off of a test result? It is vital that you answer these questions and get people in agreement BEFORE you launch a test. Success and failure of any test is not in the lift it provides, but your ability to act on that data in a clear and expedited manner.

Groups will often write these down in a document and get signoff from different leads of the organization, or create a wiki and provide access to everyone. Each group has their own unique version of this, but it is vital that before you get to the action phase, that everyone is clear on how and why you do what you do. Nothing kills a program faster than people desperately trying to stop a change because they were “wrong” or people not willing to act on data because they were not ready to go. Never start a test if you are not able to act on the results, and rules of action are your way to ensure that you will be able to do this.

Program Champion –

Outside of having a single success metric, there may be no greater determination of long term success than having someone with some power in your organization who sponsors and pushes testing. Having a person who can navigate the political waters to enable the disciplines listed above, and to help communicate and hold people accountable for results. This person may not be involved in day to day actions, but they will often hold regular review meetings, talk to different groups, and present results to senior leadership. It is not about “owning” testing, optimization is part of everyone’s job, but this person helps get testing the attention it needs and holds people accountable.

You can look at any of the best testing organizations, and without fail you can name this person instantly and clearly. That person has bought in and while the program is often not perfect, they have helped lead it in a way that there is no doubt who sponsors it, why they push it, and how much value it brings to your company.

These disciplines represent just the first steps to really becomes a optimization leader, but they are vital in order for you to be able to grow and to facilitate the adoption of more advances skills. In the second set of disciplines, we will explore ways to think about testing to insure that you are getting more value from each of the actions which you just helped facilitate.

To navigate the entire testing series:
Testing 101 / Testing 202 / Testing 303 – Part 1 / Testing 303 – Part 2

Rant – A Result is Not the Same as the Value

Just to make something clear, you will get a result from any action you take. Some good, some bad, some are neutral, but in all cases, there is a result. That result comes from not only what you do, but the context of the environment where you made it. Just slapping yourself on the back for the result you get misses the entire point. How is doing what you did, how is that any better then any other action you could have taken? If you don’t know this, then you have no clue the action value of what you chose to do. You could have just chosen the second worst of the 1000 different options you have in front of you, but because you ignore the context, you are happy, you pat yourself on the back and you tell people all about how great you are.

The goal is not to get a result, it is to become more efficient, so that you get more for what you are spending, or the same for less. Any action that doesn’t add efficiency is not adding value for your company.

History has shown that it is always the more efficient company that succeeds. That means you need two piece of information for any action: The “value” of the action (how much better it is then other alternatives) and the cost of that action. To do that, you have to be able to leverage two key things: Causal information (value) and discipline (cost) to measure the value of relative actions, and the cost to achieve them.

Correlative data, by itself, can NEVER tell you this, since it can only tell you a single rate of action of what happened in the past. It can only tell you what did happen, not what could happen, or the value of those actions. You have people who click on a link that see on average 3 pages per visit? Cool, but what about if they clicked on something else? or if that wasn’t even a possible action for them? What would they have done? would they have done more or less? Would they have seen 5 pages? 2? What about the cost to get them to see 5 pages? What about the cost to redesign that page, or the relative cost of impacting that segment versus another? Correlative data only tells you the rate of action, it is impossible for correlative data to tell you the value of those actions, since that is a comparative analysis. That difference is the only thing that matters, and you simply can’t answer it with correlative data alone…

So why then do we hear nothing but the value of this correlative data? Why do we pretend it tells us more then it does? Why do we pretend it informs decisions that it doesn’t? Why do we build fancy dashboards, or initiatives, or entire strategies on missing this fundamental point about the data you are using? It isn’t the data, it has always told you exactly the same thing, the problem is that we pretend we get so much more from it then we really do.

You can’t rely on passive action, you can’t be passive in data acquisition or you actions of that data. Passive data is nice to have, but is not enough. You have to create a cycle of using causal information to inform correlative information, not the other way around if you want real value.

Why we do what we do: Fighting fear – Loss Aversion

One of the challenges that just about any group new to testing has is trying to get buy-in and support from various other groups, usually with strong opposition from UX and branding teams, but also from just about any other group that interacts with the site. The largest reason for this is Loss Aversion, or “the disutility of giving up an object is greater than the utility associated with acquiring it.” To put simply, we get too caught up in what we lose that we miss what we gain. People fear the lack of control that opening up their ideas to analysis brings, and with that fear comes some of the biggest hurdles that programs need to overcome.

How many times have you had to try and get sign-off from a new group only to have them push back or say that doesn’t “feel right”. How many times have you wanted to bring testing to new people only to have them shy away the first moment what they were sure would win, loses? The irony of this is that some of the most ardent supporters of testing in mature programs are the very people who were challenging the programs at the very start. Anyone that has built their reputation on their artistic talent, or by declaring “here is how we are going to do this” has to overcome their fear of losing that control in order to gain the power and efficiency that testing can bring.

How many times have you gone into a conference room with people to brain storm? Or gone to an offsite just to come back with a long list of who you are going to target? Or some written rules on color guidelines or the like. There is a lot of hard work that goes into those efforts, but the problem is that they are filled with assumptions and compromise. They are designed to either make everyone feel like they contributed, or to make the HiPPO happy. What testing does in the best cases is stop that cycle, so that you are no longer trying to figure out the one way to make things work, but instead have open discussions on what is feasible. It democratizes ideas and is agnostic as to the value of them. The entire point is to be able to measure the value of each idea against the others and figure out the best one to go with. There is a great deal of fear, what if you are wrong? Does this make me look bad? My way has always been “right”, and so on. People have built empires on this fallacy, often with no one holding them accountable to the actual value of those ideas.

So how do you fight this? The first thing you must do is to get everyone to agree on what you are trying to accomplish. This has to be a single thing that you can measure and that is universal across the site (this is not about group A versus group B, this is about everyone working together to improve the site). I have seen so many programs struggle, get no value, or end up in political quagmires simply because they refused this first step. It can be very difficult or very easy, but at the end of the day, the single greatest determination of future success for optimization is agreement on what you are trying to accomplish.

Once you have that measure, then it becomes about taking those ideas and measuring them against that goal. Remember you want to challenge the common theory, meaning you should include null assumptions and things that contradict what you think will win. It serves you no good if you align everyone on finding an answer to a question if that question is irrelevant or sub optimal. What is funny is that there is almost an inverse correlation between what people think will win, and what does win. Get opinions from multiple sources, especially from one or two from outside the group that has owned that concept or portion of the site.

Step three is simply test. But at the end of the test, don’t worry if you were wrong, and don’t make it about you versus me. This is about everyone working together to find what works. If everyone is working for the same goal, then it is easy for everyone to get the credit and for everyone to align. The entire point was that the better the ideas feed into the system, the more diverse and risky those ideas are, the more you learn and the better the results you will have. You have to stop worrying about who was right and instead encourage people to be wrong. Being wrong gives you so much more than being right, and it gives you new learning to share and bring value to other parts of the site.

If you do this enough, you will get to the point where you no longer need to have those large conferences or off sites, you just need to compile the feasible options and move forward with letting the test tell you where to go. It becomes less about trying to fit the square peg into the round hole (or in some cases, into no hole) and more about aligning to move forward with what you learn. You will not end up at the feared 48 shades of blue axiom, but instead you will end up where you treat all feasible ideas as valuable. It frees up your UX and creative teams to try new things and to not worry about upsetting their superiors. It allows them the flexibility and the ability to be “wrong”.

Everyone is fearful of the unknown and the risk of giving something up. What is important is to share the challenge and the reward and to make it about adding value to what they were already doing, and to not blame anyone when they were wrong. Testing an idea is not about the loss of control, it is about helping to make it as successful as possible. This can’t be about you versus them, or my idea versus yours, in order to succeed you need everyone to work on achieving the same goal. Any system is only as good as its input, and any input without a proper system to facilitate it will always lack value in the end. Encourage new ideas, encourage trial and error, as long as you have a system in place to mitigate loss, you have so much more you can gain from learning a new path or stopping a bad practice on your site.