My last two blog posts for ConversionXL have lead to a great deal of controversial comments, which means they did their job. My goal was in no way to troll the industry or to beat down easy targets, it was instead to challenge a number of things that get held up as shields of competence. While my first article on Designer’s and their myths got plenty of fiery comments, it was the second article, on the many lies of the CRO community that really seems to have pissed people off. So much time was spent just reacting as if I was trying to flame the industry and so little time was spent actually discussing the merit of the points I raised that I wonder if this is because everyone agrees, or more likely that people seek out confirmation and not to actually grow their skills. I fully claim the writing on that one was far from my best and I will freely admit that I am super passionate about that topic (as anyone that reads these rants can attest), but I am severely disappointed in the lack of intellectual curiosity that is being shown by the audience as a whole.
No point has been beaten up more then my comment about my testing streak, which currently sits at 6 failed tests in over 5 years. People latched onto that and thought I was full of it without noticing that it was just a bullet point in a much larger topic of being ok with failed tests or accepting inefficiency in their program. Pretty much universally people dismiss my claim about the streak (despite the fact that it is 100% true) which is why I don’t actually bring it up that often, and while I do accept that on its face it is a teapot argument, the rational thing to do would be to ask if I am claiming dramatically different results then what people are getting, if I am doing things radically different then what they are doing. Instead of allowing for myopia people should be evaluating claims to see if they are the same old tired crap or if they are actually different, and then fulfilling scientific discipline by performing based on the stated criteria and seeing if they get the results that I claim.
I am not saying I am not a crackpot, I am just saying that you should see if the crazy is valuable before throwing your rotten fruit at me.
I think this also highlights one of the most disappointingly predictable parts of the industry, as people seek out echo chambers to feel better about what they are doing. Testing is the ultimate expression in dealing with uncertainty and more then anything it is about optimizing people, not websites. Because of this it can be isolating, scary, threatened and generally misunderstood, so of course people seek out comfort in their peers. It requires far more effort to look beyond what you agree with to see if there is something new out there. I still read the crap that Tim Ashe puts out, and I will read mindless marketing blogs and articles because I have to challenge the things I think I know. Isn’t that the heart of optimization anyways?
Challenge your own assumptions, not just other people’s.
As the calendar turns to 2015 most people have spent the last few weeks talking about how many amazing things happened in 2014 and marveling about how much things have changed. While the world is moving at a faster pace I am struck by how little optimization and marketing have failed to change since I came on the scene 12 years ago. While there are amazing bells and whistles and so much smoke being blown, very little has really changed about what marketers are really doing, which is a shame considering how poor most marketers have continued to be at their job.
In the past few years there has been an inundation of technology and talk about advanced marketing techniques, big data has spammed everyone to death, and so many people are promising amazing new techniques yet it is just the same old tired crap with a new shade of paint. Marketers have had access to many of these same tactics from well before I joined. Its not like BI and statistical techniques were invented in this century. In the world of online optimization there has been almost no real improvements despite the fact that the marketplace has been spammed with so many new tools and so much new attention. When I started I was working with Offermatica, which became Test&Target, which is now Target. The top “personalization” tool was Touch Clarity. Its biggest competitors are now things like monetate and optimizely, two tools that didn’t exist. Its main competitor optimost is now essentially non existent. There are now a thousand smaller tools like VWO and unbounce and even midway tools like conductrics that provide in many ways the best of both worlds. So new names on the building, but what has really changed?
In terms of data the companies come and go but you still have the same data tools that were there when I started. Now its called Google Analytics instead of urchin, or Adobe Marketing Suite instead of Omniture, but its still the same crap. People talk about all these new acquisition channels like Facebook (friendster 3.0?), mobile, twitter, social, SEO, and SEM strategies as if they aren’t the exact same discussions with a slightly different flavor that was going on long before. Yes there are many more tools to do attribution but still no way for it to really matter or be of value. Yes there are a thousand tools promising targeting of users and yet there is a near certainty that people will go with their gut and not have the discipline to get any real value from these tools; along with a near certainty of them convincing themselves and others that they are getting value. If you are just having the same discussion or thinking in terms of how to take an old strategy to a new medium, then you are not changing, you are forcing the world to stop advancing for the sake of your own cognitive dissonance. Either you treat a new tool and medium as a way to change or challenge what you know or think or you are the problem. The world gets stuck waiting for those that shouldn’t to get caught up with those who should.
I honestly had more features when I first started with an IBM netezza system for data and Offermatica back at version 13 then there exists today, not that it really mattered. Nothing matters about the tools, anyone can fail with any tool, what matters is how you change the way you think in relation to the tools. Yes we have brought the tools to the masses, in the same way that Europeans discovered America and its thousands of inhabitants, and apparently with as much damage to the landscape. So much time is wasted on trying to “improve the customer experience” or “talk to the customer” or “create a 1 on 1 connection” as if those things have any real meaning. I could walk up to 10 different marketers and ask them to really define that term and there is near a 100% chance I either get pure BS or a completely different answer from all of them. If you are still thinking in the same terms then no matter what you do you are going to get stuck with the same results. And don’t get me started on qualitative research, marketing coherts, personas or the same old tired design and UX BS. If it matters, the results will tell you, if it doesn’t, then you are wasting time and resources. That stuff has been around for a really long time and if this is the best it can produce then it obviously is a waste of time.
To make it even worse the market being inundated with the same tired posturing means that anyone new is most likely going to fall for the cotton candy advice they read or hear and that sounds good, and will never know how useless most of it is. Without the ability to really differentiate or think about things in a way that is focused on results all we really get is a negative feedback look that continues to propagate more useless BS and continue the cycle.
I think it is time that people stop talking about how much has changed and instead focused on how little has really changed. I issue a challenge to everyone to give themselves one more new year’s resolution:
I will stop being a “marketer”, I will instead focus on results and efficiency and forget titles, popular opinion, and past BS.
Just look at tools and channels and all of that with a new light. Stop trying to be the same old tired useless BS and instead of trying to copy everyone else just look at things in a new light. Challenge assumptions and “best practices”. Challenge talking heads and just try and see if the things you were doing or the things you believe get better results then other ways of tackling the same problem. Purposely break your own habits and your own perceptions and see just what you can do with each tool and each opportunity. Hell, look for new problems and try to focus your energy on getting the best results from those.
Don’t confuse action with movement and don’t confuse time passing with advancement. The technology may have more flashy buttons and new names and may get faster but the people using it who refuse to change alongside it or who allow others to convince them that they have changed when they are just rearrange deck chairs are the real challenge and the real problem. Either be different or stop pretending anything has really changed.
There are many unique parts to optimizing on a lower traffic site, but by far the most annoying is an expected high level of variance. As part of my new foray into the world of lead generation I am conducting a variance study on one of our most popular landing pages.
For those that are not clear what a variance study is, it is when you do multiple variations of the same control and you measure all of the interactions against each other. In this case I have 5 versions of control which gives you a total of 20 data points (all 5 compared to the other 4). The point of these studies is to evaluate what the normal expected variance range is as well as the minimum and maximum outcomes from the range. It is also designed to measure this over time so that you can see when and where it normalizes down to as each site and page will have a normalization curve and a normal level of variance. For a large retail site with thousands of conversions a day you can expect around 2% variance after 7-10 days. For a lead generation site with a limited product catalog and much lower numbers, you can expect higher. You will always have more variance in a visit based metric system then a visitor based metric system as you are adding the complexity of multiple interactions being treated distinctly instead of in aggregate.
There are many important outcomes to these studies. It helps you design your rules of action including needed differentiation and needed amounts of data. It helps you understand what the best measure of confidence is for your site and how actionable it is. It also helps you understand normalization curves, especially in visitor based metric systems as you can start to understand if your performance is going to normalize in 3 days or 7. Assume you will need a minimum of 6-7 days past that period for the average test to end.
The most annoying thing is understanding all the complexities of confidence and how variance can really mess it up. There are many different ways to measure confidence, from frequentest to Bayesian and P-Score to Chi Square. The most common ways are Z-test or T-Test calculations. While there are many different calculations they all generally are supposed to tell you very similar things. The most important of which is what is the likelihood that the change you are making is causing the lift you see. Higher confidence means that you are more likely to get the desired result. This means that in a perfect world a variance study should have 0% confidence and you are hoping for very low marks. The real world is rarely so kind though and knowing just how far off from that ideal is extremely important to knowing how and when to act on data.
This is what I get from my 5 experience variance study:
To clarify, this is using a normal Z-Test P-Score approach and there are over the bare minimum conversions that most people recommend (100 per experience). This is being done through Google Experiments. The highest variance I have ever dealt with on a consistent basis is 5% and anything over 3% is pretty rare. Getting an average variance of 11.83% after 5 days is just insane:
This is just not acceptable. I should not be able to get 97% confidence from forced noise. It makes any normal form of confidence almost completely meaningless. To make it worse, if I did not do this type of study or if I did not understand variance and confidence then I can easily make a false positive claim from a change. These types of errors (both type 1 and type 2) are especially dangerous because it allows people to claim an impact when there is not one and allow people to justify their opinions through purely random noise.
If you do not know your variance or do have never done a variance study, I strongly recommend that you do so. They are vital to really making functional changes to your site and will allow you to avoid wasting so much resources and times on false leads.
So much has changed in my world recently and I wanted to give everyone a heads-up. After 5+ years trying to fix some of the largest and most complicated organizational optimization issues I have stepped away from Adobe and have decided to go in a somewhat new direction. I have taken a position as Director of Optimization for a small company in the Carlsbad, CA area called Questpoint where I will be overseeing optimization of a number of lead gen situations.
What this means is that I now deal with much smaller but much more meaningful measures of success. It also means that I can now talk much more directly about the challenges I face and the solutions as they present themselves to me. I will continue to investigate the theoretical challenges of optimization but will also be more directly talking about the realities of testing on a budget. I will be using a number of tools including Google Analytics and Google Experiments and will be breaking down the advantages and disadvantages of them in comparison to the enterprise level tools that I was familiar with.
Here is to the new path before me and here is to the many barriers and hills one must climb to bring that boulder to the top of the mountain.