Category: Organization
Something New: SEO and AB Testing
As someone who has been working in the world of digital marketing for a number of years now, I’ve seen firsthand the importance of SEO and A/B testing. These two tactics are crucial for any business that wants to succeed online, and yet, I see so many companies ignoring them or not fully understanding how to utilize them effectively.
First, let’s start with SEO. Search engine optimization is the process of optimizing your website and its content to rank higher in search engine results pages (SERPs). This is important because the higher your website ranks, the more traffic you’ll receive, and the more traffic you receive, the more chances you have to convert those visitors into paying customers.
Unfortunately, SEO is often overlooked or misunderstood by many businesses. They either think it’s too complicated or they simply don’t see the value in it. This is a huge mistake. SEO is not something that you can ignore or skimp on. It’s a long-term strategy that requires consistent effort and attention.
But it’s not just about putting in the work. It’s also about understanding how search engines work and what they’re looking for when ranking websites. This includes things like keyword research, on-page optimization, and technical SEO. It’s about creating high-quality content that is both informative and engaging for users, and making sure that your website is user-friendly and easy to navigate.
Ignoring SEO can have serious consequences for your business. Not only will you miss out on potential traffic and customers, but your competitors who are investing in SEO will likely outrank you in the SERPs. This can lead to a vicious cycle where you’re constantly playing catch-up and never fully catching up.
Now, let’s talk about A/B testing. A/B testing, also known as split testing, is the process of comparing two versions of something (like a website or an email campaign) to see which one performs better. This can be anything from the color of a button to the layout of a page.
A/B testing is crucial because it allows you to make informed decisions about your marketing efforts. It helps you understand what works and what doesn’t, and it allows you to optimize your campaigns for maximum effectiveness.
Unfortunately, A/B testing is often neglected or not done correctly. Many businesses either don’t realize the importance of it or they don’t have the resources or know-how to set up and run A/B tests. This can lead to poor decision-making and wasted resources.
To properly conduct an A/B test, you need to make sure you have a clear idea of what you are changing and a large enough sample size to get reliable results. You also need to make sure you’re testing one variable at a time and not making any other changes to the campaign during the test. It’s also important to let the test run for a sufficient amount of time to get accurate results.
But even when businesses do A/B testing, they often make the mistake of not acting on the results. Maybe they don’t like the outcome of the test, or maybe they just don’t have the resources to implement the changes. Either way, this is a huge mistake. A/B testing is only valuable if you use the results to make improvements and optimize your campaigns.
So, why do I see so many businesses ignoring or not fully utilizing SEO and A/B testing? I think it comes down to a lack of understanding and resources. SEO and A/B testing can be time-consuming and require a certain level of expertise. It’s easy for businesses to get caught up in the day-to-day tasks and not make the time or invest in the resources necessary to focus on these important tactics.
Data, Optimization, and Growth – The 3 Sides of the Product Management Pyramid
Its been a while since I last wrote for this blog, and in that time I have moved away from straight optimization into a more central Growth role, focused on data, optimization, and product managing core systems. The one thing that unifies all these disciplines is the one core mission statement:
“Use data and added information to improve the efficiency of resources towards the bottom line for the company.”
Or to put even more simply:
“Make the organization work smarter, not harder.”
The issue of course is that you have a multitude of people in an organization, all of which think their work is perfect and everyone else is the problem. Fingers get pointed all the time, some real, most not, due to cognitive dissonance and the stories that we convince ourselves of to justify our efforts. The truth is that most people are little more then blind moles digging through the preverbal dirt, and when they manage to not get crushed by a rock we tell ourselves that we are masters of our domain.
So, when someone comes in and tries to say, hey, maybe don’t do what you are doing or tackle a problem a different way, they react with all the grace and thoughtfulness of a rabid dog cornered by a pack of small cats. This is especially true in the product management world where so much is made of stories about users, or success, or trying to come up with some giant master idea that will solve all the problems in the world.
There are no magical ideas.
There are no silver bullets.
Yet that group, along with Design, seem so focused on ideas that they miss the forest for the trees. Success and failure isn’t based on a specific idea. Yes, there are better ideas then others, but the situation that those ideas are born into have so much more to do with overall success then the idea. The best idea in the world acted on poorly is going to have worse outcomes (See every redesign ever done), then the most mediocre idea acted on correctly.
With that in mind I wanted to take a moment and discuss what actually matters from a Product focused view of the business world, and how data can and should be a part of that conversation. Instead of just listing all the things to not do, and there are many, many, many books (easiest first read is The Halo Effect) on that subject, I want to instead focus on 3 core disciplines that will make any organization or Product group effective.
Scope trumps Ideas
The ability to properly scope and phase a project is the single most important part of any new initiative. If you put too much into an idea and you don’t have proper measures in place, then you are just wasting resources and become even more invested in its outcome. Under scope a project and you may see a change in behavior but no idea what, if any, of it is noise versus real outcomes.
Being able to properly create an MVP, being able to properly measure and be disciplined in going forward with that first or any phase, determines both the efficiency of the resources as well as the long-term success of a project or idea.
Scoping is the Product Managers super power. Failure to do it and you are a 4year old child trying to lift a car. Do it properly and you are the Hulk lifting a mountain.
So why then do so many product managers suck at this skill? Part of it is that most product managers start out in other disciplines, all of which focus on small pieces of the big picture. You may come from sales (scope is a completed sale), engineering (did it get done, yet or no), design (does it look pretty, yes or no), or a thousand other places. Scoping is a skill that is unique to the practice that most people have never spent more then a few seconds thinking about. On top of that you are always under pressure to sell an idea or to show progress, which means checking off boxes becomes important even when getting results should matter.
The other key part here is that scoping properly ensures that you have proper tracking towards a bottom line and a clear measure of success. Any phase of a project that doesn’t have a clear start and stop and a clear specific measure of bottom line (Not Texas Sharpshooter based) success is doomed. Its that simple. Every part of the human psyche and the inherent reward system in most organizations is designed to make the people involved believe or tell others there is success, even when there is not.
Being able to identify what is the MVP, what is the natural small phases, what the go to market and what the decision points are going into a project are what is going to give it success. It also allows you to plan to resources and not try to plan resources to projects. Knowing what you are working with and what the decision point is allows you complete control to ensure you can make proper decisions.
Discipline Wins
To steal a line from one of the most overused “thought leaders” of our time:
“Its not the 1 thing you say yes to, it’s the thousand things you say no to”
We live in a world with finite resources and unlimited expectations of results. In that setting you need to make sure that you are maximizing the use of your resources and making sure that the right things get finished. The fastest way to stop that is to start pulling resources left, right and away from every project whenever a passing fancy gets into anyone’s head.
And to be clear, it doesn’t matter if that idea comes from the janitor, the designer, the engineer, or the CEO, all ideas must be prioritized and acted on correctly. You are going to have way more success attacking 3 projects properly scoped then 15 projects done on a whim. Even when you factor in the higher chances of success from fragility and high beta, the fact that things are not focused and not meeting minimum viable status means that even when something randomly does work, you wont be able to exploit that information due to all of the problems you have added to the system.
A list of completed tasks is nice, except the number of tasks has nothing to do with the value or outcomes of those tasks. I could do 500 tasks in a week and accomplish nothing. I could not finish a single task in that week and still be highly impactful to the company (though you probably need to break up your task lists). Utilization is a measure of man hours, not of value.
And to be clear, no one person should own the complete specifics of what passes and what doesn’t, that should be up to a smaller very highly informed group that represent different POVs throughout the organization. What that one person can own however is stopping all ideas and being a gate keeper until they have been properly vetted and scoped and until everyone agrees on priority.
Of course this is easier said then done, as not only do people, especially those that are attempting to climb a corporate ladder, have inverse correlation from outcomes to pleasing their bosses, but all ideas can be made to sound good, no matter how foolish they may actually be. Not only that but even if you have 20 great ideas, if you are not managing resources and balancing the needs of the business, then you may go too far down a specific rabbit hole with no way to recover.
Making your boss happy helps you, but not scrambling and not giving in to every whim, no matter how much you want to, gets you results. Like all else, what matters most to you, personal success vs. company success is upt
Balance of Vision is Critical
I was lucky enough to meet and talk with Marissa Mayer way back in the day (Pre-Yahoo) and she gave me the single best piece of advice for a product manager. The job is to equally meet 3 goals. Where you want the product to go, where the market wants the product to go, and where the market needs the product to go. If you go too far down one side (only respond to pressure, only go where you sales teams want to go, only following your own vision) you will fail in the long run.
That advice still holds true very much today. You need to balance the core infrastructure, the need for short term growth, and the need for long term growth. No matter what pressure comes your way, no matter what resources you have, no matter who is screaming or how much your sales people are saying you need feature X or to build product Y, the same discipline and focus is needed.
Don’t focus on your infrastructure, and cost and maintenance will break everything. This means you need to know where in your scoping to make the infrastructure from MVP to robust. You have to build in plans and direct measurements of where and how you are going to move to a more robust infrastructure and make sure you stay disciplined enough to not go beyond that point. Technical debt is a bitch to fix.
Focus too much on adding features for short term success (like what your sales people may ask for) and your product will lose its actual value and will lose focus on what made it successful in the first place. Measure and make sure that you get short term wins, but adding technical debt or losing focus on core drive means you may end up with a fancy looking glob of uselessness.
Focus too much on your own vision, and you won’t have the correct pieces in place or be able to adapt to market trends. Add decision points at every step to make sure to validate and try to disprove your own vision.
Each of those steps have different measurement methods and goals. Each of those must be measured correctly to make sure you are not just doing work to do work. Each one of those need people trying to disprove the common thought process to be successful. None of them work if you are just using data to validate someone’s ego.
So you have to know how to scope and where to spend resources. You need to be able to stop reacting to everything and to make sure that you have focus and the correct measurement to succeed. You also need to make sure you have the right people fulfilling the correct roles. You need to have an independent analysis team that is not beholden to your own drive and ego. You need to make sure you have a sales team trying to get people to have the best short-term success. You need a technical team who isn’t trying to please everyone and who is trying to build a robust infrastructure. You must have a team that can act fast and test assumptions. You must have a project management team who is not just filling out forms but holding people accountable to scope, phases and timelines.
These things can and often do break. Product Management is the mastering and managing of all of them. There are no magic bullets and the specifics of each of these core focuses is dependent on the situation, but its focusing and trying to master each one that will determine your long term success. Its not some shiny idea or pleasing your boss or the CEO, its about doing the small ugly things well, consistently, and with laser like focus that will determine long term success or failure of the product and team.
It’s the discipline of balancing all these visions and moving people together. Data is just the tool to add a sanity check at each phase.
Cauterizing Open Wounds
One of the most difficult parts of starting your own program or of consulting with a new organization is the need to evaluate and change existing practices. In almost all cases groups have been optimizing for a while, often times with one or more people owning the program and who have built their reputations off of prior practice. Any prior actions have been done with their name attached and they have enjoyed the perceptions of success. The problem is though that people rarely evaluate the reality of their statements and are often not aware or too busy to really know if what they are saying is real or pure BS (this explains the entire agency system).
This can be extremely problematic as it is vital to stop any bad practices before you can implement needed discipline and really make a positive impact for your company. It does you no good to look into things like fragility or efficiency, or in controlled experiments or segment discovery if you are operating in a world where people expect to test out 1 or 2 ideas based on opinions and to do this in 2-3 days. If your organization actually thinks that things like 48 hours to run 8 tests and clicks on a button are a measure of success then no amount of real optimization is going to matter until you make it clear just how off the entire process is. Of course if you do this poorly then you are just making yourself public enemy number 1 and since you are the new guy in the room you are basically setting yourself up for failure.
The key is to understand the issues and tackle all of them without prejudice and to evaluate the program for all of them. That way people see that you are not attacking someone or something but simply evaluating the program for inefficiencies. If everything is up for grabs and somethings pass and something go then at the least you are removing the direct confrontational element from it. If you can further push the conversation into one of what defines success and simply focus on those components then many of the would be battles simple fall by the wayside.
Generally the things that need to evaluated and often changed fall into a number of common categories. These include:
Acting on test:
-
False belief in confidence
Acting too quickly
No consistent rules of action
Lack of Process:
-
No consistent way of getting results live
No single person owning test ideation, just random ideas thrown up
Lack of data control:
-
Wrong metrics
No variance study
Lack of proper segment analysis
The main problem with any or all of these is that there will be a library of tests that people have believed and most likely built entire strategies around. It doesn’t matter if it is what pages do or do not work, the impact of certain changes or where and who to test to, this misinformation is far more damaging then any positive result that you could generate.
All results are contextual, and as such this means that you must set the proper context in order to really evaluate the impact of a test or process. If you have people believing a 200% increase because they were looking at one group and on clicks on a button then it can be nearly impossible to talk about a 5% RPV increase because it just sounds too small and not as important to them, despite the fact that the 200% click increase could have actually caused a 10% loss in revenue. If you or others do not understand the core principles and math involved then they are more likely to fall for any BS that they come across. You must focus on education and on the disciplines, not just stories if you want to make meaningful long term impact.
This is why stopping the bleeding is such an important and difficult task to overcome. People don’t realize how far off they really are and often times have never been called out for their BS, resulting in entire careers built on bad outcomes and false conclusions. In my case I am looking at everything from acting too quickly (18 conversions versus 32 conversions is meaningless), a lack of variance understanding, and a lack of discipline on test ideas. These things were not done because someone was malicious or self serving. they were not done because of a lack of intelligence or a lack of want to improve the business, they were simply done because the person did not know better and because there is just so much bad information out there.
The real challenge here is controlling expectations and helping people understand the error in their ways. I am extremely lucky to work with a number of very smart people who are willing to listen to and understand issues which they never knew they were dealing with, like the variance problems I previously discussed. The challenge if far more in people understand that just because they come from a place that is used to testing in 1-2 days or in tracking a certain thing it just means that they were really good at wasting their companies time and resources. It is also important to also set proper expectations on what the movement speed will be. If they are thinking you can get a result in 2-3 days and it is going to take 2-3 weeks, this can completely shift your view of optimization to a the negative despite the fact that you are really moving from something that was damaging the company to something that is going to cause consistent positive growth.
More then anything it is important to realize that you have to stop all bleeding and make that the primary focus before you can overly concern yourself with making big changes. This doesn’t mean that you don’t do any tests or the like, in fact it is important for people to see what they should be doing so that they can really appreciate how far off they were prior. If someone doesn’t know what success looks like then any point on the map can be success for them. It simply means that controlling the message and focusing on education is vital at the start of any program.
One Problem but Many Voices – The One Thing People Need to Understand about Optimization
The hardest challenge when working with different groups in the optimization space is often trying to get past their misconceptions and to help them view optimization in a different form. It doesn’t matter if they have been doing testing for 1 day or 10 years, there is still a massive difference in efficiency and the value that can be generated. Results are not random yet so many believe they are because they misunderstand optimization on the most fundamental levels. The reality of real successful optimization is often far from the perceived reality from those just entering the space. The number of misconceptions is so large that it can often be nearly impossible to prioritize them or to tackle them all.
Because this problem is so common, I reached out to the smartest people I know in the industry and asked them to share their thoughts about what the one thing they wished people understood about optimization.
Rhett Norton – Consultant
Without discipline companies go through the motions of testing without ever really achieving amazing long term results. The most successful companies I’ve worked with have been successful with creating discipline in parts of their testing program. I’ve never seen a company that is disciplined in every aspect of optimization, but hey, maybe your company could be the first.
Drew Phillips – Consultant
It is free form in that you need to have the flexibility to optimize elements that you find to be influential, not lock yourself into a specific roadmap. Optimization is a process that changes as you learn from each campaign. You will get the most out of your optimization efforts by iterating off of things you learn from previous tests.
Brandon Anderson – Consultant
Sometimes organizations that have been doing A/B testing for years feel like they need to work on complex activities in order to continue progressing. My experience is that even mature organizations need to look past the hype of new and shiny buzzwords and determine which activities will give them the highest efficiency. Get the 80% with 20% of the effort by focusing on the basics.
Ryan Roberts – Solution Architect
I also wish people were more careful about how they read test results. People that rely solely on confidence calculations are going to end up with a lot more wrong conclusions than they think. They need to understand what the rules of conclusive results should be for their site. And they have to apply them religiously to each test they run.
Doug Mumford – Consultant
While there are some tests that will require more time a lot of highly valuable tests can be done with three lines of CSS or jQuery, loaded up in four browsers to make sure everything looks good (and perhaps an iPhone and iPad), and launch. Have a bias for action.
If I had to characterize my own answer to the question it would be that there is a massive difference between action and value. Just running a test, be it one or 500, is not the mark that you are successfully optimizing. Optimization is about how you tackle large assumptions, and about how you act on data, and even how you think about what data can and can’t tell you. So much time is wasted in the pursuit of executing on assumptions and against the propagation of agendas which is the exact opposite of where the value of optimization comes from.
It is about discipline, and statistics, and variance, and technical solutions, and dealing with senior management and dealing with biases and assumptions. It is all that and more. It is a means to an ends, but that end is increased revenue for your organization, not just blindly reaching an audience or making an individual look good. The more you try to justify a specific action or the more complicated you make something, the less value you get and the more time you waste. Just understanding that action in and of itself is not the answer is the first step to being truly open to solving the largest challenges that optimization programs face. The challenge is never in running tests, the real challenge is finding solutions and ways to even have these conversations.
What do you find as the one thing you wished people understood about optimization? What are you doing to solve it?
TL;DR 