Category: Organization
Data, Optimization, and Growth – The 3 Sides of the Product Management Pyramid
Its been a while since I last wrote for this blog, and in that time I have moved away from straight optimization into a more central Growth role, focused on data, optimization, and product managing core systems. The one thing that unifies all these disciplines is the one core mission statement:
“Use data and added information to improve the efficiency of resources towards the bottom line for the company.”
Or to put even more simply:
“Make the organization work smarter, not harder.”
The issue of course is that you have a multitude of people in an organization, all of which think their work is perfect and everyone else is the problem. Fingers get pointed all the time, some real, most not, due to cognitive dissonance and the stories that we convince ourselves of to justify our efforts. The truth is that most people are little more then blind moles digging through the preverbal dirt, and when they manage to not get crushed by a rock we tell ourselves that we are masters of our domain.
So, when someone comes in and tries to say, hey, maybe don’t do what you are doing or tackle a problem a different way, they react with all the grace and thoughtfulness of a rabid dog cornered by a pack of small cats. This is especially true in the product management world where so much is made of stories about users, or success, or trying to come up with some giant master idea that will solve all the problems in the world.
There are no magical ideas.
There are no silver bullets.
Yet that group, along with Design, seem so focused on ideas that they miss the forest for the trees. Success and failure isn’t based on a specific idea. Yes, there are better ideas then others, but the situation that those ideas are born into have so much more to do with overall success then the idea. The best idea in the world acted on poorly is going to have worse outcomes (See every redesign ever done), then the most mediocre idea acted on correctly.
With that in mind I wanted to take a moment and discuss what actually matters from a Product focused view of the business world, and how data can and should be a part of that conversation. Instead of just listing all the things to not do, and there are many, many, many books (easiest first read is The Halo Effect) on that subject, I want to instead focus on 3 core disciplines that will make any organization or Product group effective.
Scope trumps Ideas
The ability to properly scope and phase a project is the single most important part of any new initiative. If you put too much into an idea and you don’t have proper measures in place, then you are just wasting resources and become even more invested in its outcome. Under scope a project and you may see a change in behavior but no idea what, if any, of it is noise versus real outcomes.
Being able to properly create an MVP, being able to properly measure and be disciplined in going forward with that first or any phase, determines both the efficiency of the resources as well as the long-term success of a project or idea.
Scoping is the Product Managers super power. Failure to do it and you are a 4year old child trying to lift a car. Do it properly and you are the Hulk lifting a mountain.
So why then do so many product managers suck at this skill? Part of it is that most product managers start out in other disciplines, all of which focus on small pieces of the big picture. You may come from sales (scope is a completed sale), engineering (did it get done, yet or no), design (does it look pretty, yes or no), or a thousand other places. Scoping is a skill that is unique to the practice that most people have never spent more then a few seconds thinking about. On top of that you are always under pressure to sell an idea or to show progress, which means checking off boxes becomes important even when getting results should matter.
The other key part here is that scoping properly ensures that you have proper tracking towards a bottom line and a clear measure of success. Any phase of a project that doesn’t have a clear start and stop and a clear specific measure of bottom line (Not Texas Sharpshooter based) success is doomed. Its that simple. Every part of the human psyche and the inherent reward system in most organizations is designed to make the people involved believe or tell others there is success, even when there is not.
Being able to identify what is the MVP, what is the natural small phases, what the go to market and what the decision points are going into a project are what is going to give it success. It also allows you to plan to resources and not try to plan resources to projects. Knowing what you are working with and what the decision point is allows you complete control to ensure you can make proper decisions.
Discipline Wins
To steal a line from one of the most overused “thought leaders” of our time:
“Its not the 1 thing you say yes to, it’s the thousand things you say no to”
We live in a world with finite resources and unlimited expectations of results. In that setting you need to make sure that you are maximizing the use of your resources and making sure that the right things get finished. The fastest way to stop that is to start pulling resources left, right and away from every project whenever a passing fancy gets into anyone’s head.
And to be clear, it doesn’t matter if that idea comes from the janitor, the designer, the engineer, or the CEO, all ideas must be prioritized and acted on correctly. You are going to have way more success attacking 3 projects properly scoped then 15 projects done on a whim. Even when you factor in the higher chances of success from fragility and high beta, the fact that things are not focused and not meeting minimum viable status means that even when something randomly does work, you wont be able to exploit that information due to all of the problems you have added to the system.
A list of completed tasks is nice, except the number of tasks has nothing to do with the value or outcomes of those tasks. I could do 500 tasks in a week and accomplish nothing. I could not finish a single task in that week and still be highly impactful to the company (though you probably need to break up your task lists). Utilization is a measure of man hours, not of value.
And to be clear, no one person should own the complete specifics of what passes and what doesn’t, that should be up to a smaller very highly informed group that represent different POVs throughout the organization. What that one person can own however is stopping all ideas and being a gate keeper until they have been properly vetted and scoped and until everyone agrees on priority.
Of course this is easier said then done, as not only do people, especially those that are attempting to climb a corporate ladder, have inverse correlation from outcomes to pleasing their bosses, but all ideas can be made to sound good, no matter how foolish they may actually be. Not only that but even if you have 20 great ideas, if you are not managing resources and balancing the needs of the business, then you may go too far down a specific rabbit hole with no way to recover.
Making your boss happy helps you, but not scrambling and not giving in to every whim, no matter how much you want to, gets you results. Like all else, what matters most to you, personal success vs. company success is upt
Balance of Vision is Critical
I was lucky enough to meet and talk with Marissa Mayer way back in the day (Pre-Yahoo) and she gave me the single best piece of advice for a product manager. The job is to equally meet 3 goals. Where you want the product to go, where the market wants the product to go, and where the market needs the product to go. If you go too far down one side (only respond to pressure, only go where you sales teams want to go, only following your own vision) you will fail in the long run.
That advice still holds true very much today. You need to balance the core infrastructure, the need for short term growth, and the need for long term growth. No matter what pressure comes your way, no matter what resources you have, no matter who is screaming or how much your sales people are saying you need feature X or to build product Y, the same discipline and focus is needed.
Don’t focus on your infrastructure, and cost and maintenance will break everything. This means you need to know where in your scoping to make the infrastructure from MVP to robust. You have to build in plans and direct measurements of where and how you are going to move to a more robust infrastructure and make sure you stay disciplined enough to not go beyond that point. Technical debt is a bitch to fix.
Focus too much on adding features for short term success (like what your sales people may ask for) and your product will lose its actual value and will lose focus on what made it successful in the first place. Measure and make sure that you get short term wins, but adding technical debt or losing focus on core drive means you may end up with a fancy looking glob of uselessness.
Focus too much on your own vision, and you won’t have the correct pieces in place or be able to adapt to market trends. Add decision points at every step to make sure to validate and try to disprove your own vision.
Each of those steps have different measurement methods and goals. Each of those must be measured correctly to make sure you are not just doing work to do work. Each one of those need people trying to disprove the common thought process to be successful. None of them work if you are just using data to validate someone’s ego.
So you have to know how to scope and where to spend resources. You need to be able to stop reacting to everything and to make sure that you have focus and the correct measurement to succeed. You also need to make sure you have the right people fulfilling the correct roles. You need to have an independent analysis team that is not beholden to your own drive and ego. You need to make sure you have a sales team trying to get people to have the best short-term success. You need a technical team who isn’t trying to please everyone and who is trying to build a robust infrastructure. You must have a team that can act fast and test assumptions. You must have a project management team who is not just filling out forms but holding people accountable to scope, phases and timelines.
These things can and often do break. Product Management is the mastering and managing of all of them. There are no magic bullets and the specifics of each of these core focuses is dependent on the situation, but its focusing and trying to master each one that will determine your long term success. Its not some shiny idea or pleasing your boss or the CEO, its about doing the small ugly things well, consistently, and with laser like focus that will determine long term success or failure of the product and team.
It’s the discipline of balancing all these visions and moving people together. Data is just the tool to add a sanity check at each phase.
Cauterizing Open Wounds
One of the most difficult parts of starting your own program or of consulting with a new organization is the need to evaluate and change existing practices. In almost all cases groups have been optimizing for a while, often times with one or more people owning the program and who have built their reputations off of prior practice. Any prior actions have been done with their name attached and they have enjoyed the perceptions of success. The problem is though that people rarely evaluate the reality of their statements and are often not aware or too busy to really know if what they are saying is real or pure BS (this explains the entire agency system).
This can be extremely problematic as it is vital to stop any bad practices before you can implement needed discipline and really make a positive impact for your company. It does you no good to look into things like fragility or efficiency, or in controlled experiments or segment discovery if you are operating in a world where people expect to test out 1 or 2 ideas based on opinions and to do this in 2-3 days. If your organization actually thinks that things like 48 hours to run 8 tests and clicks on a button are a measure of success then no amount of real optimization is going to matter until you make it clear just how off the entire process is. Of course if you do this poorly then you are just making yourself public enemy number 1 and since you are the new guy in the room you are basically setting yourself up for failure.
The key is to understand the issues and tackle all of them without prejudice and to evaluate the program for all of them. That way people see that you are not attacking someone or something but simply evaluating the program for inefficiencies. If everything is up for grabs and somethings pass and something go then at the least you are removing the direct confrontational element from it. If you can further push the conversation into one of what defines success and simply focus on those components then many of the would be battles simple fall by the wayside.
Generally the things that need to evaluated and often changed fall into a number of common categories. These include:
Acting on test:
-
False belief in confidence
Acting too quickly
No consistent rules of action
Lack of Process:
-
No consistent way of getting results live
No single person owning test ideation, just random ideas thrown up
Lack of data control:
-
Wrong metrics
No variance study
Lack of proper segment analysis
The main problem with any or all of these is that there will be a library of tests that people have believed and most likely built entire strategies around. It doesn’t matter if it is what pages do or do not work, the impact of certain changes or where and who to test to, this misinformation is far more damaging then any positive result that you could generate.
All results are contextual, and as such this means that you must set the proper context in order to really evaluate the impact of a test or process. If you have people believing a 200% increase because they were looking at one group and on clicks on a button then it can be nearly impossible to talk about a 5% RPV increase because it just sounds too small and not as important to them, despite the fact that the 200% click increase could have actually caused a 10% loss in revenue. If you or others do not understand the core principles and math involved then they are more likely to fall for any BS that they come across. You must focus on education and on the disciplines, not just stories if you want to make meaningful long term impact.
This is why stopping the bleeding is such an important and difficult task to overcome. People don’t realize how far off they really are and often times have never been called out for their BS, resulting in entire careers built on bad outcomes and false conclusions. In my case I am looking at everything from acting too quickly (18 conversions versus 32 conversions is meaningless), a lack of variance understanding, and a lack of discipline on test ideas. These things were not done because someone was malicious or self serving. they were not done because of a lack of intelligence or a lack of want to improve the business, they were simply done because the person did not know better and because there is just so much bad information out there.
The real challenge here is controlling expectations and helping people understand the error in their ways. I am extremely lucky to work with a number of very smart people who are willing to listen to and understand issues which they never knew they were dealing with, like the variance problems I previously discussed. The challenge if far more in people understand that just because they come from a place that is used to testing in 1-2 days or in tracking a certain thing it just means that they were really good at wasting their companies time and resources. It is also important to also set proper expectations on what the movement speed will be. If they are thinking you can get a result in 2-3 days and it is going to take 2-3 weeks, this can completely shift your view of optimization to a the negative despite the fact that you are really moving from something that was damaging the company to something that is going to cause consistent positive growth.
More then anything it is important to realize that you have to stop all bleeding and make that the primary focus before you can overly concern yourself with making big changes. This doesn’t mean that you don’t do any tests or the like, in fact it is important for people to see what they should be doing so that they can really appreciate how far off they were prior. If someone doesn’t know what success looks like then any point on the map can be success for them. It simply means that controlling the message and focusing on education is vital at the start of any program.
One Problem but Many Voices – The One Thing People Need to Understand about Optimization
The hardest challenge when working with different groups in the optimization space is often trying to get past their misconceptions and to help them view optimization in a different form. It doesn’t matter if they have been doing testing for 1 day or 10 years, there is still a massive difference in efficiency and the value that can be generated. Results are not random yet so many believe they are because they misunderstand optimization on the most fundamental levels. The reality of real successful optimization is often far from the perceived reality from those just entering the space. The number of misconceptions is so large that it can often be nearly impossible to prioritize them or to tackle them all.
Because this problem is so common, I reached out to the smartest people I know in the industry and asked them to share their thoughts about what the one thing they wished people understood about optimization.
Rhett Norton – Consultant
Without discipline companies go through the motions of testing without ever really achieving amazing long term results. The most successful companies I’ve worked with have been successful with creating discipline in parts of their testing program. I’ve never seen a company that is disciplined in every aspect of optimization, but hey, maybe your company could be the first.
Drew Phillips – Consultant
It is free form in that you need to have the flexibility to optimize elements that you find to be influential, not lock yourself into a specific roadmap. Optimization is a process that changes as you learn from each campaign. You will get the most out of your optimization efforts by iterating off of things you learn from previous tests.
Brandon Anderson – Consultant
Sometimes organizations that have been doing A/B testing for years feel like they need to work on complex activities in order to continue progressing. My experience is that even mature organizations need to look past the hype of new and shiny buzzwords and determine which activities will give them the highest efficiency. Get the 80% with 20% of the effort by focusing on the basics.
Ryan Roberts – Solution Architect
I also wish people were more careful about how they read test results. People that rely solely on confidence calculations are going to end up with a lot more wrong conclusions than they think. They need to understand what the rules of conclusive results should be for their site. And they have to apply them religiously to each test they run.
Doug Mumford – Consultant
While there are some tests that will require more time a lot of highly valuable tests can be done with three lines of CSS or jQuery, loaded up in four browsers to make sure everything looks good (and perhaps an iPhone and iPad), and launch. Have a bias for action.
If I had to characterize my own answer to the question it would be that there is a massive difference between action and value. Just running a test, be it one or 500, is not the mark that you are successfully optimizing. Optimization is about how you tackle large assumptions, and about how you act on data, and even how you think about what data can and can’t tell you. So much time is wasted in the pursuit of executing on assumptions and against the propagation of agendas which is the exact opposite of where the value of optimization comes from.
It is about discipline, and statistics, and variance, and technical solutions, and dealing with senior management and dealing with biases and assumptions. It is all that and more. It is a means to an ends, but that end is increased revenue for your organization, not just blindly reaching an audience or making an individual look good. The more you try to justify a specific action or the more complicated you make something, the less value you get and the more time you waste. Just understanding that action in and of itself is not the answer is the first step to being truly open to solving the largest challenges that optimization programs face. The challenge is never in running tests, the real challenge is finding solutions and ways to even have these conversations.
What do you find as the one thing you wished people understood about optimization? What are you doing to solve it?
When Heuristics go Bad – Dealing with Common Optimization Practices – Part 2
My first trip through the common heuristics of conversion rate optimization looked at two of the more common testing ideas and how they usually reach false or limiting conclusions. In my second part I want to look at general testing theory best practices and how they can be major limiting factors in the success of your program.
It is important to remember that you are always going to get an outcome so this is not about can you make money. How you and the people in your organization think about testing is the largest factor in what you value that optimization produces. This is an evaluation of the efficiency of the method and how much does it produce for the same or less resources. In concept you can spend infinite amount of resources to achieve any end goal, but the reality is that we are always faced with a finite amount of time and population, which means we must always be looking for ways to improve inefficient systems. If we continue to be limited by these common heuristics then the industry as a whole will continue to produce minimal results compared to what it can and should be producing.
Always have a Hypothesis –
There is not more misunderstood term then hypothesis. In all likelihood it is because most are familiar only with their 6th grade (at least in my school) science instruction or they took classroom formal science in college. In those fields we operate like we have unlimited time and resources and we are trying to validate whether a drug will cause cancer, not whether a banner will get more clicks if it is blue or red. The stakes are higher and the models are much more simple in classroom controlled studies for cancer. There is a lot to scientific method, especially when approached from a resource efficiency perspective that is not considered in such a simplistic view of idea validation.
We must apply scientific rigor, but we must also make sure that all actions make sense in real world situations, which means that efficiency and minimizing regret are more important than validation of an individual’s opinion. It is not that scientific method relies on the use of a hypothesis, it is simply that we mistake a hypothesis with a correct hypothesis; we seek validation for our opinions and not the discovery of the best way to proceed. Science is also about proving one idea versus all other alternative hypothesis yet we ignore that part of the discipline because it is not the part that allows someone to see if they are right. In the grand scheme of things we are drastically over valuing test ideas and that is distracting from the parts of the process that provide value.
Let’s start with the basics. You should never, and I mean never, run a test if you do not have a single success metric for your entire site. In most cases this is to make more money, but whatever it is, this goal exists outside of the concept of the test. You must also must have rigid measurement and action rules that are reproducible, which means that you must understand real world situations like the limitations of confidence and variance.
You can then have an opinion about what you think will happen when you make a change. The problem is when we confuse that opinion with the measured goals of the test. Even worse we limit what we compare resulting in massively inefficient use of your time and effort. Just because you believe that improving your navigation will get people to spend more time on your site, that is completely irrelevant to the end goal of making more money. Your belief that more engagement will result in more revenue is not enough to make it so. If you are right AND if that also produces more revenue, then you will know that from revenue. If you are wrong you will only know that from revenue. We must construct our actions to produce answers to our opinion and to what is best for our organization. Hypothesis and ideas are just a very small part of a much more complex and important picture, and over focus on them allows people to avoid the responsibility and the benefit on focusing on all those other parts, which are the ones that really make a difference over time for any and all testing programs.
The worst factor of this is that it allows people to fall for congruence bias and to fail to ask the right questions. We become so used to the conversation around a single idea that the concept of discovery and challenging assumptions is more word then action. Questions can be incredibly important to the success of a program, but only if they are tackled in the right order and used to focus attention, not as the final validation of spent attention. If your hypothesis is that a certain navigation change will result in more engagement, then the correct use of your resources are either which of a number of different versions of the navigation will produce the most revenue or if you can, which section on your site produces the most engagement when changed. In both cases you have adapted your “hypothesis” to present a more efficient and functional use of your time. The hypothesis exists, but it is not the constraint of the test. If you are right, you will see it. If you are wrong, you will make more money.
This means that having a hypothesis is important, but only if it is not the test charter. Have an idea what you are trying to accomplish and make sure that you go about seeing the value of certain actions compared to each other is more important. Sometimes the most effective hypothesis are “I believe that we do not know the value of different sections on our pages.” Don’t confuse your opinion on what will win with a successful test. Challenge assumptions and design efforts to maximize what you can do with what you have and you will never be without opinions. The best answers are always when you are proven wrong, but if you get too caught up on validating your hypothesis, then you will always be missing the largest lessons you could be learning.
We need to optimize X because it is losing Y
This is the classic problem of confusing rate and value, or more correctly correlative and causal inference. We confuse what we want to happen with what is really happening. Just because people were doing X and now they are doing Y, it doesn’t mean that this is directly causing any change, positive or negative to our end goals. Outside of the three rules of proving causation the real issue here is that we get tied to our beliefs about a pattern of events even when the data cannot possibly validate that conclusion. Understanding and acting on what you know as opposed to what you want to have happen is the difference between being data driven and simply being data justified.
Think about it this way, I have 23% clicks on one section of my page and 0% on another. If I were to improve one of those which one is going to produce the biggest returns? The answer here is that you do not know. A rate of interaction cannot possibly tell you the value of changing that item. Some of the most important parts of any user experience are things that can’t even be clicked.
This plays out outside of clicks too. We have a product funnel and we see more people leaving on page 3, therefore we need to test on page 3. The reality is that more or less people may or may not be tied to more or less revenue. Even if it is tied it may be a qualification issue higher, or a user interaction issue, or simply too many people in a prior step. This is called a linear assumption fallacy, where we assume that when we have 5 people and 2 convert that if we have 10 people 4 will convert. Linear models are rare in nature but are easy to understand, so we fall back on comfort over realistic understanding.
The act of figuring out what to test can be difficult but it is never improved by pretending we have validation of our own ideas when we have nothing to justify them. We need to be open to discovering where we should go and to focus on some set path. In almost all cases you will find that you are wrong, often dramatically so, about where problems really are and how to fix them. This is why it is so important to not try and focus solely on more or less correlative actions. We can and should be able to test fast enough and with few enough resources that we will never be limited to this realm unless we can are stuck there mentally.
Like so much else what you spend your time and effort on is incredibly important. There are a thousand things you can improve and there are always new ideas. Justifying them falsely or focusing on them instead of the discipline of testing is nothing but a drag on your entire testing program. Test ideation is about 1% of the value derived from a test program yet it is 90%+ of where people like to spend their time. A 5% gain that took 2 months is worth a lot less than a 10% gain that took 2 weeks. The most important issues we must face are not about generating test ideas or validating our beliefs about how to improve our site, it is about discovering and applying resources to make sure that we are doing the 10% option and not the 5% option. If we overly focus on test ideas and not the discipline of applying them correctly we are never going to going to achieve what should be achieved. If we get lost trying to focus only on where we want to go, then you will always be limited in the possible outcomes you can generate.
TL;DR 