Rant: Rants, Streaks, and the Lack of Intellectual Curiosity

My last two blog posts for ConversionXL have lead to a great deal of controversial comments, which means they did their job. My goal was in no way to troll the industry or to beat down easy targets, it was instead to challenge a number of things that get held up as shields of competence. While my first article on Designer’s and their myths got plenty of fiery comments, it was the second article, on the many lies of the CRO community that really seems to have pissed people off. So much time was spent just reacting as if I was trying to flame the industry and so little time was spent actually discussing the merit of the points I raised that I wonder if this is because everyone agrees, or more likely that people seek out confirmation and not to actually grow their skills. I fully claim the writing on that one was far from my best and I will freely admit that I am super passionate about that topic (as anyone that reads these rants can attest), but I am severely disappointed in the lack of intellectual curiosity that is being shown by the audience as a whole.

No point has been beaten up more then my comment about my testing streak, which currently sits at 6 failed tests in over 5 years. People latched onto that and thought I was full of it without noticing that it was just a bullet point in a much larger topic of being ok with failed tests or accepting inefficiency in their program. Pretty much universally people dismiss my claim about the streak (despite the fact that it is 100% true) which is why I don’t actually bring it up that often, and while I do accept that on its face it is a teapot argument, the rational thing to do would be to ask if I am claiming dramatically different results then what people are getting, if I am doing things radically different then what they are doing. Instead of allowing for myopia people should be evaluating claims to see if they are the same old tired crap or if they are actually different, and then fulfilling scientific discipline by performing based on the stated criteria and seeing if they get the results that I claim.

I am not saying I am not a crackpot, I am just saying that you should see if the crazy is valuable before throwing your rotten fruit at me.

I think this also highlights one of the most disappointingly predictable parts of the industry, as people seek out echo chambers to feel better about what they are doing. Testing is the ultimate expression in dealing with uncertainty and more then anything it is about optimizing people, not websites. Because of this it can be isolating, scary, threatened and generally misunderstood, so of course people seek out comfort in their peers. It requires far more effort to look beyond what you agree with to see if there is something new out there. I still read the crap that Tim Ashe puts out, and I will read mindless marketing blogs and articles because I have to challenge the things I think I know. Isn’t that the heart of optimization anyways?

Challenge your own assumptions, not just other people’s.


Rant – Things: They Keep on Changing?

As the calendar turns to 2015 most people have spent the last few weeks talking about how many amazing things happened in 2014 and marveling about how much things have changed. While the world is moving at a faster pace I am struck by how little optimization and marketing have failed to change since I came on the scene 12 years ago. While there are amazing bells and whistles and so much smoke being blown, very little has really changed about what marketers are really doing, which is a shame considering how poor most marketers have continued to be at their job.

In the past few years there has been an inundation of technology and talk about advanced marketing techniques, big data has spammed everyone to death, and so many people are promising amazing new techniques yet it is just the same old tired crap with a new shade of paint. Marketers have had access to many of these same tactics from well before I joined. Its not like BI and statistical techniques were invented in this century. In the world of online optimization there has been almost no real improvements despite the fact that the marketplace has been spammed with so many new tools and so much new attention. When I started I was working with Offermatica, which became Test&Target, which is now Target. The top “personalization” tool was Touch Clarity. Its biggest competitors are now things like monetate and optimizely, two tools that didn’t exist. Its main competitor optimost is now essentially non existent. There are now a thousand smaller tools like VWO and unbounce and even midway tools like conductrics that provide in many ways the best of both worlds. So new names on the building, but what has really changed?

In terms of data the companies come and go but you still have the same data tools that were there when I started. Now its called Google Analytics instead of urchin, or Adobe Marketing Suite instead of Omniture, but its still the same crap. People talk about all these new acquisition channels like Facebook (friendster 3.0?), mobile, twitter, social, SEO, and SEM strategies as if they aren’t the exact same discussions with a slightly different flavor that was going on long before. Yes there are many more tools to do attribution but still no way for it to really matter or be of value. Yes there are a thousand tools promising targeting of users and yet there is a near certainty that people will go with their gut and not have the discipline to get any real value from these tools; along with a near certainty of them convincing themselves and others that they are getting value. If you are just having the same discussion or thinking in terms of how to take an old strategy to a new medium, then you are not changing, you are forcing the world to stop advancing for the sake of your own cognitive dissonance. Either you treat a new tool and medium as a way to change or challenge what you know or think or you are the problem. The world gets stuck waiting for those that shouldn’t to get caught up with those who should.

I honestly had more features when I first started with an IBM netezza system for data and Offermatica back at version 13 then there exists today, not that it really mattered. Nothing matters about the tools, anyone can fail with any tool, what matters is how you change the way you think in relation to the tools. Yes we have brought the tools to the masses, in the same way that Europeans discovered America and its thousands of inhabitants, and apparently with as much damage to the landscape. So much time is wasted on trying to “improve the customer experience” or “talk to the customer” or “create a 1 on 1 connection” as if those things have any real meaning. I could walk up to 10 different marketers and ask them to really define that term and there is near a 100% chance I either get pure BS or a completely different answer from all of them. If you are still thinking in the same terms then no matter what you do you are going to get stuck with the same results. And don’t get me started on qualitative research, marketing coherts, personas or the same old tired design and UX BS. If it matters, the results will tell you, if it doesn’t, then you are wasting time and resources. That stuff has been around for a really long time and if this is the best it can produce then it obviously is a waste of time.

To make it even worse the market being inundated with the same tired posturing means that anyone new is most likely going to fall for the cotton candy advice they read or hear and that sounds good, and will never know how useless most of it is. Without the ability to really differentiate or think about things in a way that is focused on results all we really get is a negative feedback look that continues to propagate more useless BS and continue the cycle.

I think it is time that people stop talking about how much has changed and instead focused on how little has really changed. I issue a challenge to everyone to give themselves one more new year’s resolution:

I will stop being a “marketer”, I will instead focus on results and efficiency and forget titles, popular opinion, and past BS.

Just look at tools and channels and all of that with a new light. Stop trying to be the same old tired useless BS and instead of trying to copy everyone else just look at things in a new light. Challenge assumptions and “best practices”. Challenge talking heads and just try and see if the things you were doing or the things you believe get better results then other ways of tackling the same problem. Purposely break your own habits and your own perceptions and see just what you can do with each tool and each opportunity. Hell, look for new problems and try to focus your energy on getting the best results from those.

Don’t confuse action with movement and don’t confuse time passing with advancement. The technology may have more flashy buttons and new names and may get faster but the people using it who refuse to change alongside it or who allow others to convince them that they have changed when they are just rearrange deck chairs are the real challenge and the real problem. Either be different or stop pretending anything has really changed.

Why we do what we do: Not everything is a nail – Maslow’s Hammer

At any given time you will often hear me quoting many famous quotes about anything and everything; that is how I relate to new experiences by trying to tie them to some bit of knowledge that I had already picked up. Probably my most common refrain lately is the famous Mike Tyson quote, “Everyone has a plan until they get punched in the face.” It’s true that everyone talks doing the right thing and everyone wants things to succeed, but as soon as there is some challenge to the prevailing world view or as soon as a small bump in the road exists, people often revert back to what they know best and turn back in on themselves. Unfortunately this is especially problematic in the business world as the only way to move forward is to change behaviors and tackle existing problems in new ways. Even more distressing is that as people fall back to what they are most comfortable with they turn towards their own disciplines and their own previous experience, limiting the ability for people of disparate talents and backgrounds to work together.

One of the things that defines people is the concept of viewing the world through their own experiences, and the most powerful experiences that we have in the modern world is our professions. Be it marketing, or engineering, management or data, we all view the world through the lens of the things we do and the challenges that we face day to day. We view the challenge of improving numbers by looking to “dialogue with our customers” or “increase efficiency through data analysis” or by “building better tools and a better user experience”. All of these in isolation seem like and often are very good ideas except when they cloud our ability to prioritize and to focus on a single outcome. Each day in the business world is really a Sisyphean climb to the top and each time that boulder rolls back on us we run back to that which we are most comfortable with. This is especially dangerous when we do not even have true accountability for the tie between those concepts and the functional bottom line outcome that we need to generate.

Abraham Maslow is famous for many things, from his hierarchy of needs to his many contributions to modern psychology. What he is often not associated with is a quote that almost everyone is familiar with, “if all you have is a hammer, everything looks like a nail.” We are all carrying hammers in the form of our world views and our professional disciplines. The key is to accept that there are many things outside of what we accept as “true” about the way to do things and about how to tackle problems. Even more when we do get evidence that does not directly correlate with our existing world view we can not dismiss it or try to understand it through that same tired lens.

Optimization at its core is the act of adding accountability to these world views and about challenging assumptions. It is about taking the existing practices of the entire organization and standing them on their end, shaking them, and finding all the holes and least effective parts. It does this not maliciously but as a mutual benefit to everyone to add a different point of view on the functions and actions that they are taking. This is why the discipline of testing is about everything but test ideas. It is about building rational rules of action and building out alternative hypothesis. This is why you focus on efficiency and multiple options and not just on what won and not about what won elsewhere or about some great idea someone had. It is why it is about patterns and not about some artificial reasoning why something won. You serve the organization a great discipline when all you do is regurgitate the nail back to someone so that they can then hit it with the same tired hammer. Optimization is the act of putting any idea and discipline through a system that allows for it to get better and for everyone to learn and to get better results.

At the same time, it is important to understand that everyone else is viewing the world through a very different lens. They are trying to tie their past experiences with new actions and new results. A marketer has always thought in terms of a dialogue with a certain user or a certain persona. That mental model has gotten where they are today. When you come in and show that there might be more effective ways to look at those same users or the the concept of personalization most likely will not work the way they envision, you are creating a very powerful form of cognitive dissonance and you are forcing people outside of that hammer that they so readily wield. Too much and you will cause major push back and possibly form an ongoing barrier to success. Too little push and you are just confirming their biases and not providing any assistance.

The key in this and in all actions is to be firm on discipline but flexible on tactics. Work with the concepts and push them past their existing barriers. This is why it is so vital to not focus on test ideas when building out a successful test. Talk about what people were already focusing on and how best you can test out that concept against many others. You want to do personalization, great, here is how we take what you were doing and serve that and other concepts to everyone. If you are right, we get to see that and if you are wrong then we found something that is better. In reality there is no downside to performance when we tackle a problem that way. It is about reaching the ends, not about the means that get you there.

Another key to this is to get people to vote on what they think will win for each test. If you do this enough and with enough varied options and you will be amazed at just how bad people are at guessing the right answer. In the last 9 tests that I have done we have averaged 8 options for each test, with some variants coming from the team, some from myself, but a large many simply expressions of the various directions that are feasible. I have asked a large team to pick there favorite and second favorite. In those 9 tests, we have had exactly 1 second place vote for all of the winners combined, and the only reason that the option got that vote was because my very talented designer picked up on the pattern and voted her least favorite. The shock of where we are versus where people thought we would be and the impact to the bottom line (over 200% improvement) has helped open doors to new ways of tackling problems, and it has done so organically.

In both tactics you are giving people the chance to tie their world view in with the results and letting them have a stake in the outcome. You are welcoming that hammer they wield but helping them see that there are many different nails to hit.

Keep in mind however that you are just as guilty as they are. Spend too much time in the world of optimization and you will start to feel like no one has any idea what they are doing and that all ideas are going to fail. It is even more important for you to challenge yourself and for you to go beyond your comfort zone in where you let testing going. Make sure you include ideas from others as much as possible, even if you are sure they are not going to work. Make sure you tie optimization in on actions that you feel might not comfortable or worth your time. Remember that the smarter someone is, the more likely they are to be impacted by biases and that you serve no good to the organization or yourself if you are not more vigilant against your own biases then you are against others.

Cauterizing Open Wounds

One of the most difficult parts of starting your own program or of consulting with a new organization is the need to evaluate and change existing practices. In almost all cases groups have been optimizing for a while, often times with one or more people owning the program and who have built their reputations off of prior practice. Any prior actions have been done with their name attached and they have enjoyed the perceptions of success. The problem is though that people rarely evaluate the reality of their statements and are often not aware or too busy to really know if what they are saying is real or pure BS (this explains the entire agency system).

This can be extremely problematic as it is vital to stop any bad practices before you can implement needed discipline and really make a positive impact for your company. It does you no good to look into things like fragility or efficiency, or in controlled experiments or segment discovery if you are operating in a world where people expect to test out 1 or 2 ideas based on opinions and to do this in 2-3 days. If your organization actually thinks that things like 48 hours to run 8 tests and clicks on a button are a measure of success then no amount of real optimization is going to matter until you make it clear just how off the entire process is. Of course if you do this poorly then you are just making yourself public enemy number 1 and since you are the new guy in the room you are basically setting yourself up for failure.

The key is to understand the issues and tackle all of them without prejudice and to evaluate the program for all of them. That way people see that you are not attacking someone or something but simply evaluating the program for inefficiencies. If everything is up for grabs and somethings pass and something go then at the least you are removing the direct confrontational element from it. If you can further push the conversation into one of what defines success and simply focus on those components then many of the would be battles simple fall by the wayside.

Generally the things that need to evaluated and often changed fall into a number of common categories. These include:

Acting on test:

    False belief in confidence
    Acting too quickly
    No consistent rules of action

Lack of Process:

    No consistent way of getting results live
    No single person owning test ideation, just random ideas thrown up

Lack of data control:

    Wrong metrics
    No variance study
    Lack of proper segment analysis

The main problem with any or all of these is that there will be a library of tests that people have believed and most likely built entire strategies around. It doesn’t matter if it is what pages do or do not work, the impact of certain changes or where and who to test to, this misinformation is far more damaging then any positive result that you could generate.

All results are contextual, and as such this means that you must set the proper context in order to really evaluate the impact of a test or process. If you have people believing a 200% increase because they were looking at one group and on clicks on a button then it can be nearly impossible to talk about a 5% RPV increase because it just sounds too small and not as important to them, despite the fact that the 200% click increase could have actually caused a 10% loss in revenue. If you or others do not understand the core principles and math involved then they are more likely to fall for any BS that they come across. You must focus on education and on the disciplines, not just stories if you want to make meaningful long term impact.

This is why stopping the bleeding is such an important and difficult task to overcome. People don’t realize how far off they really are and often times have never been called out for their BS, resulting in entire careers built on bad outcomes and false conclusions. In my case I am looking at everything from acting too quickly (18 conversions versus 32 conversions is meaningless), a lack of variance understanding, and a lack of discipline on test ideas. These things were not done because someone was malicious or self serving. they were not done because of a lack of intelligence or a lack of want to improve the business, they were simply done because the person did not know better and because there is just so much bad information out there.

The real challenge here is controlling expectations and helping people understand the error in their ways. I am extremely lucky to work with a number of very smart people who are willing to listen to and understand issues which they never knew they were dealing with, like the variance problems I previously discussed. The challenge if far more in people understand that just because they come from a place that is used to testing in 1-2 days or in tracking a certain thing it just means that they were really good at wasting their companies time and resources. It is also important to also set proper expectations on what the movement speed will be. If they are thinking you can get a result in 2-3 days and it is going to take 2-3 weeks, this can completely shift your view of optimization to a the negative despite the fact that you are really moving from something that was damaging the company to something that is going to cause consistent positive growth.

More then anything it is important to realize that you have to stop all bleeding and make that the primary focus before you can overly concern yourself with making big changes. This doesn’t mean that you don’t do any tests or the like, in fact it is important for people to see what they should be doing so that they can really appreciate how far off they were prior. If someone doesn’t know what success looks like then any point on the map can be success for them. It simply means that controlling the message and focusing on education is vital at the start of any program.