The life of working with organizations to bring about change is a difficult and frustrating one, but at the same time there are some truly amazing moments. The times you see an organization change and truly look at optimization differently, when they get past misconceptions and “best practices” are truly amazing. While much of my writing focuses on the dark side of this world, I did want to take some time and share a few small stories from the front-line that are what to me consulting is all about.

Story #1: I was working with a hospitality and travel company where I was hoping to come in and change a number of practices that may not have been beneficial to the company. Starting out an example test series to work through discover and exploit, they decided to run an inclusion/exclusion test on a page template for their destination landing pages. We discovered that their main content block section was having a negative impact to their page and that removing it would result in 12% lift to RPV. We also discovered that a small section on their right rail that they never even thought about was actually the most influential part of the page.

We were able to do this with very little effort (basic CSS), but because we challenged a number of internal assumptions, we learned about things they never considered, reduced maintenance cost, and achieved a major lift to their bottom line. The real kicker of this testing was that it was both the lowest resource test they had ever run and their most valuable.

Story #2: Working with another travel/hospitality organization, I came in because they had not been able to get any momentum with their testing due to a previous belief that it required too much resources. We were able to leverage their existing infrastructure and ran a simple inclusion/exclusion test on one of their landing pages. We did this all in one conversation without any pretext. In this case we discovered that almost every element individually was negative to the page, meaning that as a whole the total was far less then the sum of the parts. We run a simple follow-up and discovered that they could generate 3-8 million dollars by simply removing one of their main offer parts, and that they could generate 4.5-12 million if they were willing to create a dynamic experience and show one of two sections based on new and returning users.

I love these types of moments because you have so many good things come out of very small actions. We now have a group that is amazed by the power of optimization. We now have a group that sees that a number of their assumptions have been proven incorrect about what works, or even for whom. We were able to generate massive fiscal impact to the business, with extremely low effort, and were able to give them insight into other opportunities to do the same thing. They were able to learn about the disciplines of testing, find out that conversion rate and revenue are not correlated, and were able to think about their entire user experience differently.

Story #3: Working with a major online retailer, they were just re-introducing optimization to their organization. They had already decided on what they were going to test, how, and when. They then jumped into their test, but I gave them guidance to not over react and to understand how contextual changes are going to play out (very large initial climb and then narrow down to almost nothing quickly). They were so excited when they had a 40+% lift in a couple of days with 99% confidence, but I asked them to wait and talked to them about what confidence really means. They held off and within the next 7 days, the results of the test were less then 2% to the point it had no impact.

Now normally you don’t like outcomes that do not generate lift, what made my day is that they stopped themselves from rushing around claiming a result that had no basis in reality. If they had ran to tell the rest of the organization about that 40+% lift, they would have done irrecoverable damage to the company as others would have attempted to do the same thing and reach the same false conclusions. Because they waited, they get to understand the nature of statistics, contextual versus spatial changes, and started thinking more about the true disciplines of optimization. Since that time they have now expanded their testing to multiple countries and are testing at an accelerated pace and driving real value throughout the organization.

There are so many moments in working with different groups where you can get frustrated, or give up, or even worse give in and tell someone what they want to hear. Equally there are many moments where you can create a false impression of impact and fake numbers for your own gain. The moments that are truly amazing are those that prove that you don’t have to do any of those things, that you can achieve amazing results, quickly, and for far less time and effort then someone might otherwise think. All it takes is to think about and tackle problems wrong, and the results that any group can achieve are truly amazing.