Data, Optimization, and Growth – The 3 Sides of the Product Management Pyramid

Its been a while since I last wrote for this blog, and in that time I have moved away from straight optimization into a more central Growth role, focused on data, optimization, and product managing core systems.   The one thing that unifies all these disciplines is the one core mission statement:

“Use data and added information to improve the efficiency of resources towards the bottom line for the company.”

Or to put even more simply:

“Make the organization work smarter, not harder.”

The issue of course is that you have a multitude of people in an organization, all of which think their work is perfect and everyone else is the problem.   Fingers get pointed all the time, some real, most not, due to cognitive dissonance and the stories that we convince ourselves of to justify our efforts.   The truth is that most people are little more then blind moles digging through the preverbal dirt, and when they manage to not get crushed by a rock we tell ourselves that we are masters of our domain.

So, when someone comes in and tries to say, hey, maybe don’t do what you are doing or tackle a problem a different way, they react with all the grace and thoughtfulness of a rabid dog cornered by a pack of small cats.  This is especially true in the product management world where so much is made of stories about users, or success, or trying to come up with some giant master idea that will solve all the problems in the world.

There are no magical ideas.

There are no silver bullets.

Yet that group, along with Design, seem so focused on ideas that they miss the forest for the trees.   Success and failure isn’t based on a specific idea.  Yes, there are better ideas then others, but the situation that those ideas are born into have so much more to do with overall success then the idea.  The best idea in the world acted on poorly is going to have worse outcomes (See every redesign ever done), then the most mediocre idea acted on correctly.

With that in mind I wanted to take a moment and discuss what actually matters from a Product focused view of the business world, and how data can and should be a part of that conversation.  Instead of just listing all the things to not do, and there are many, many, many books (easiest first read is The Halo Effect) on that subject, I want to instead focus on 3 core disciplines that will make any organization or Product group effective.

Scope trumps Ideas

The ability to properly scope and phase a project is the single most important part of any new initiative.   If you put too much into an idea and you don’t have proper measures in place, then you are just wasting resources and become even more invested in its outcome.  Under scope a project and you may see a change in behavior but no idea what, if any, of it is noise versus real outcomes.

Being able to properly create an MVP, being able to properly measure and be disciplined in going forward with that first or any phase, determines both the efficiency of the resources as well as the long-term success of a project or idea.

Scoping is the Product Managers super power.  Failure to do it and you are a 4year old child trying to lift a car.  Do it properly and you are the Hulk lifting a mountain.

So why then do so many product managers suck at this skill?   Part of it is that most product managers start out in other disciplines, all of which focus on small pieces of the big picture.  You may come from sales (scope is a completed sale), engineering (did it get done, yet or no), design (does it look pretty, yes or no), or a thousand other places.   Scoping is a skill that is unique to the practice that most people have never spent more then a few seconds thinking about.   On top of that you are always under pressure to sell an idea or to show progress, which means checking off boxes becomes important even when getting results should matter.

The other key part here is that scoping properly ensures that you have proper tracking towards a bottom line and a clear measure of success.  Any phase of a project that doesn’t have a clear start and stop and a clear specific measure of bottom line (Not Texas Sharpshooter based) success is doomed.  Its that simple.   Every part of the human psyche and the inherent reward system in most organizations is designed to make the people involved believe or tell others there is success, even when there is not.

Being able to identify what is the MVP, what is the natural small phases, what the go to market and what the decision points are going into a project are what is going to give it success.   It also allows you to plan to resources and not try to plan resources to projects.   Knowing what you are working with and what the decision point is allows you complete control to ensure you can make proper decisions.

Discipline Wins

To steal a line from one of the most overused “thought leaders” of our time:

“Its not the 1 thing you say yes to, it’s the thousand things you say no to”

We live in a world with finite resources and unlimited expectations of results.   In that setting you need to make sure that you are maximizing the use of your resources and making sure that the right things get finished.  The fastest way to stop that is to start pulling resources left, right and away from every project whenever a passing fancy gets into anyone’s head.

And to be clear, it doesn’t matter if that idea comes from the janitor, the designer, the engineer, or the CEO, all ideas must be prioritized and acted on correctly.   You are going to have way more success attacking 3 projects properly scoped then 15 projects done on a whim.   Even when you factor in the higher chances of success from fragility and high beta, the fact that things are not focused and not meeting minimum viable status means that even when something randomly does work, you wont be able to exploit that information due to all of the problems you have added to the system.

A list of completed tasks is nice, except the number of tasks has nothing to do with the value or outcomes of those tasks.  I could do 500 tasks in a week and accomplish nothing.  I could not finish a single task in that week and still be highly impactful to the company (though you probably need to break up your task lists).   Utilization is a measure of man hours, not of value.

And to be clear, no one person should own the complete specifics of what passes and what doesn’t, that should be up to a smaller very highly informed group that represent different POVs throughout the organization.  What that one person can own however is stopping all ideas and being a gate keeper until they have been properly vetted and scoped and until everyone agrees on priority.

Of course this is easier said then done, as not only do people, especially those that are attempting to climb a corporate ladder, have inverse correlation from outcomes to pleasing their bosses, but all ideas can be made to sound good, no matter how foolish they may actually be.  Not only that but even if you have 20 great ideas, if you are not managing resources and balancing the needs of the business, then you may go too far down a specific rabbit hole with no way to recover.

Making your boss happy helps you, but not scrambling and not giving in to every whim, no matter how much you want to, gets you results.  Like all else, what matters most to you, personal success vs. company success is upt

 

Balance of Vision is Critical

 

I was lucky enough to meet and talk with Marissa Mayer way back in the day (Pre-Yahoo) and she gave me the single best piece of advice for a product manager.   The job is to equally meet 3 goals.  Where you want the product to go, where the market wants the product to go, and where the market needs the product to go.   If you go too far down one side (only respond to pressure, only go where you sales teams want to go, only following your own vision) you will fail in the long run.

That advice still holds true very much today.  You need to balance the core infrastructure, the need for short term growth, and the need for long term growth.  No matter what pressure comes your way, no matter what resources you have, no matter who is screaming or how much your sales people are saying you need feature X or to build product Y, the same discipline and focus is needed.

Don’t focus on your infrastructure, and cost and maintenance will break everything.   This means you need to know where in your scoping to make the infrastructure from MVP to robust.   You have to build in plans and direct measurements of where and how you are going to move to a more robust infrastructure and make sure you stay disciplined enough to not go beyond that point.  Technical debt is a bitch to fix.

Focus too much on adding features for short term success (like what your sales people may ask for) and your product will lose its actual value and will lose focus on what made it successful in the first place.   Measure and make sure that you get short term wins, but adding technical debt or losing focus on core drive means you may end up with a fancy looking glob of uselessness.

Focus too much on your own vision, and you won’t have the correct pieces in place or be able to adapt to market trends.   Add decision points at every step to make sure to validate and try to disprove your own vision.

Each of those steps have different measurement methods and goals.  Each of those must be measured correctly to make sure you are not just doing work to do work.  Each one of those need people trying to disprove the common thought process to be successful.  None of them work if you are just using data to validate someone’s ego.

So you have to know how to scope and where to spend resources.  You need to be able to stop reacting to everything and to make sure that you have focus and the correct measurement to succeed.  You also need to make sure you have the right people fulfilling the correct roles.  You need to have an independent analysis team that is not beholden to your own drive and ego.  You need to make sure you have a sales team trying to get people to have the best short-term success.  You need a technical team who isn’t trying to please everyone and who is trying to build a robust infrastructure.  You must have a team that can act fast and test assumptions.  You must have a project management team who is not just filling out forms but holding people accountable to scope, phases and timelines.

These things can and often do break.  Product Management is the mastering and managing of all of them.  There are no magic bullets and the specifics of each of these core focuses is dependent on the situation, but its focusing and trying to master each one that will determine your long term success.   Its not some shiny idea or pleasing your boss or the CEO, its about doing the small ugly things well, consistently, and with laser like focus that will determine long term success or failure of the product and team.

It’s the discipline of balancing all these visions and moving people together.  Data is just the tool to add a sanity check at each phase.

Advertisement

Rant: Rants, Streaks, and the Lack of Intellectual Curiosity

My last two blog posts for ConversionXL have lead to a great deal of controversial comments, which means they did their job. My goal was in no way to troll the industry or to beat down easy targets, it was instead to challenge a number of things that get held up as shields of competence. While my first article on Designer’s and their myths got plenty of fiery comments, it was the second article, on the many lies of the CRO community that really seems to have pissed people off. So much time was spent just reacting as if I was trying to flame the industry and so little time was spent actually discussing the merit of the points I raised that I wonder if this is because everyone agrees, or more likely that people seek out confirmation and not to actually grow their skills. I fully claim the writing on that one was far from my best and I will freely admit that I am super passionate about that topic (as anyone that reads these rants can attest), but I am severely disappointed in the lack of intellectual curiosity that is being shown by the audience as a whole.

No point has been beaten up more then my comment about my testing streak, which currently sits at 6 failed tests in over 5 years. People latched onto that and thought I was full of it without noticing that it was just a bullet point in a much larger topic of being ok with failed tests or accepting inefficiency in their program. Pretty much universally people dismiss my claim about the streak (despite the fact that it is 100% true) which is why I don’t actually bring it up that often, and while I do accept that on its face it is a teapot argument, the rational thing to do would be to ask if I am claiming dramatically different results then what people are getting, if I am doing things radically different then what they are doing. Instead of allowing for myopia people should be evaluating claims to see if they are the same old tired crap or if they are actually different, and then fulfilling scientific discipline by performing based on the stated criteria and seeing if they get the results that I claim.

I am not saying I am not a crackpot, I am just saying that you should see if the crazy is valuable before throwing your rotten fruit at me.

I think this also highlights one of the most disappointingly predictable parts of the industry, as people seek out echo chambers to feel better about what they are doing. Testing is the ultimate expression in dealing with uncertainty and more then anything it is about optimizing people, not websites. Because of this it can be isolating, scary, threatened and generally misunderstood, so of course people seek out comfort in their peers. It requires far more effort to look beyond what you agree with to see if there is something new out there. I still read the crap that Tim Ashe puts out, and I will read mindless marketing blogs and articles because I have to challenge the things I think I know. Isn’t that the heart of optimization anyways?

Challenge your own assumptions, not just other people’s.

Rant – Things: They Keep on Changing?

As the calendar turns to 2015 most people have spent the last few weeks talking about how many amazing things happened in 2014 and marveling about how much things have changed. While the world is moving at a faster pace I am struck by how little optimization and marketing have failed to change since I came on the scene 12 years ago. While there are amazing bells and whistles and so much smoke being blown, very little has really changed about what marketers are really doing, which is a shame considering how poor most marketers have continued to be at their job.

In the past few years there has been an inundation of technology and talk about advanced marketing techniques, big data has spammed everyone to death, and so many people are promising amazing new techniques yet it is just the same old tired crap with a new shade of paint. Marketers have had access to many of these same tactics from well before I joined. Its not like BI and statistical techniques were invented in this century. In the world of online optimization there has been almost no real improvements despite the fact that the marketplace has been spammed with so many new tools and so much new attention. When I started I was working with Offermatica, which became Test&Target, which is now Target. The top “personalization” tool was Touch Clarity. Its biggest competitors are now things like monetate and optimizely, two tools that didn’t exist. Its main competitor optimost is now essentially non existent. There are now a thousand smaller tools like VWO and unbounce and even midway tools like conductrics that provide in many ways the best of both worlds. So new names on the building, but what has really changed?

In terms of data the companies come and go but you still have the same data tools that were there when I started. Now its called Google Analytics instead of urchin, or Adobe Marketing Suite instead of Omniture, but its still the same crap. People talk about all these new acquisition channels like Facebook (friendster 3.0?), mobile, twitter, social, SEO, and SEM strategies as if they aren’t the exact same discussions with a slightly different flavor that was going on long before. Yes there are many more tools to do attribution but still no way for it to really matter or be of value. Yes there are a thousand tools promising targeting of users and yet there is a near certainty that people will go with their gut and not have the discipline to get any real value from these tools; along with a near certainty of them convincing themselves and others that they are getting value. If you are just having the same discussion or thinking in terms of how to take an old strategy to a new medium, then you are not changing, you are forcing the world to stop advancing for the sake of your own cognitive dissonance. Either you treat a new tool and medium as a way to change or challenge what you know or think or you are the problem. The world gets stuck waiting for those that shouldn’t to get caught up with those who should.

I honestly had more features when I first started with an IBM netezza system for data and Offermatica back at version 13 then there exists today, not that it really mattered. Nothing matters about the tools, anyone can fail with any tool, what matters is how you change the way you think in relation to the tools. Yes we have brought the tools to the masses, in the same way that Europeans discovered America and its thousands of inhabitants, and apparently with as much damage to the landscape. So much time is wasted on trying to “improve the customer experience” or “talk to the customer” or “create a 1 on 1 connection” as if those things have any real meaning. I could walk up to 10 different marketers and ask them to really define that term and there is near a 100% chance I either get pure BS or a completely different answer from all of them. If you are still thinking in the same terms then no matter what you do you are going to get stuck with the same results. And don’t get me started on qualitative research, marketing coherts, personas or the same old tired design and UX BS. If it matters, the results will tell you, if it doesn’t, then you are wasting time and resources. That stuff has been around for a really long time and if this is the best it can produce then it obviously is a waste of time.

To make it even worse the market being inundated with the same tired posturing means that anyone new is most likely going to fall for the cotton candy advice they read or hear and that sounds good, and will never know how useless most of it is. Without the ability to really differentiate or think about things in a way that is focused on results all we really get is a negative feedback look that continues to propagate more useless BS and continue the cycle.

I think it is time that people stop talking about how much has changed and instead focused on how little has really changed. I issue a challenge to everyone to give themselves one more new year’s resolution:

I will stop being a “marketer”, I will instead focus on results and efficiency and forget titles, popular opinion, and past BS.

Just look at tools and channels and all of that with a new light. Stop trying to be the same old tired useless BS and instead of trying to copy everyone else just look at things in a new light. Challenge assumptions and “best practices”. Challenge talking heads and just try and see if the things you were doing or the things you believe get better results then other ways of tackling the same problem. Purposely break your own habits and your own perceptions and see just what you can do with each tool and each opportunity. Hell, look for new problems and try to focus your energy on getting the best results from those.

Don’t confuse action with movement and don’t confuse time passing with advancement. The technology may have more flashy buttons and new names and may get faster but the people using it who refuse to change alongside it or who allow others to convince them that they have changed when they are just rearrange deck chairs are the real challenge and the real problem. Either be different or stop pretending anything has really changed.

Why we do what we do: Not everything is a nail – Maslow’s Hammer

At any given time you will often hear me quoting many famous quotes about anything and everything; that is how I relate to new experiences by trying to tie them to some bit of knowledge that I had already picked up. Probably my most common refrain lately is the famous Mike Tyson quote, “Everyone has a plan until they get punched in the face.” It’s true that everyone talks doing the right thing and everyone wants things to succeed, but as soon as there is some challenge to the prevailing world view or as soon as a small bump in the road exists, people often revert back to what they know best and turn back in on themselves. Unfortunately this is especially problematic in the business world as the only way to move forward is to change behaviors and tackle existing problems in new ways. Even more distressing is that as people fall back to what they are most comfortable with they turn towards their own disciplines and their own previous experience, limiting the ability for people of disparate talents and backgrounds to work together.

One of the things that defines people is the concept of viewing the world through their own experiences, and the most powerful experiences that we have in the modern world is our professions. Be it marketing, or engineering, management or data, we all view the world through the lens of the things we do and the challenges that we face day to day. We view the challenge of improving numbers by looking to “dialogue with our customers” or “increase efficiency through data analysis” or by “building better tools and a better user experience”. All of these in isolation seem like and often are very good ideas except when they cloud our ability to prioritize and to focus on a single outcome. Each day in the business world is really a Sisyphean climb to the top and each time that boulder rolls back on us we run back to that which we are most comfortable with. This is especially dangerous when we do not even have true accountability for the tie between those concepts and the functional bottom line outcome that we need to generate.

Abraham Maslow is famous for many things, from his hierarchy of needs to his many contributions to modern psychology. What he is often not associated with is a quote that almost everyone is familiar with, “if all you have is a hammer, everything looks like a nail.” We are all carrying hammers in the form of our world views and our professional disciplines. The key is to accept that there are many things outside of what we accept as “true” about the way to do things and about how to tackle problems. Even more when we do get evidence that does not directly correlate with our existing world view we can not dismiss it or try to understand it through that same tired lens.

Optimization at its core is the act of adding accountability to these world views and about challenging assumptions. It is about taking the existing practices of the entire organization and standing them on their end, shaking them, and finding all the holes and least effective parts. It does this not maliciously but as a mutual benefit to everyone to add a different point of view on the functions and actions that they are taking. This is why the discipline of testing is about everything but test ideas. It is about building rational rules of action and building out alternative hypothesis. This is why you focus on efficiency and multiple options and not just on what won and not about what won elsewhere or about some great idea someone had. It is why it is about patterns and not about some artificial reasoning why something won. You serve the organization a great discipline when all you do is regurgitate the nail back to someone so that they can then hit it with the same tired hammer. Optimization is the act of putting any idea and discipline through a system that allows for it to get better and for everyone to learn and to get better results.

At the same time, it is important to understand that everyone else is viewing the world through a very different lens. They are trying to tie their past experiences with new actions and new results. A marketer has always thought in terms of a dialogue with a certain user or a certain persona. That mental model has gotten where they are today. When you come in and show that there might be more effective ways to look at those same users or the the concept of personalization most likely will not work the way they envision, you are creating a very powerful form of cognitive dissonance and you are forcing people outside of that hammer that they so readily wield. Too much and you will cause major push back and possibly form an ongoing barrier to success. Too little push and you are just confirming their biases and not providing any assistance.

The key in this and in all actions is to be firm on discipline but flexible on tactics. Work with the concepts and push them past their existing barriers. This is why it is so vital to not focus on test ideas when building out a successful test. Talk about what people were already focusing on and how best you can test out that concept against many others. You want to do personalization, great, here is how we take what you were doing and serve that and other concepts to everyone. If you are right, we get to see that and if you are wrong then we found something that is better. In reality there is no downside to performance when we tackle a problem that way. It is about reaching the ends, not about the means that get you there.

Another key to this is to get people to vote on what they think will win for each test. If you do this enough and with enough varied options and you will be amazed at just how bad people are at guessing the right answer. In the last 9 tests that I have done we have averaged 8 options for each test, with some variants coming from the team, some from myself, but a large many simply expressions of the various directions that are feasible. I have asked a large team to pick there favorite and second favorite. In those 9 tests, we have had exactly 1 second place vote for all of the winners combined, and the only reason that the option got that vote was because my very talented designer picked up on the pattern and voted her least favorite. The shock of where we are versus where people thought we would be and the impact to the bottom line (over 200% improvement) has helped open doors to new ways of tackling problems, and it has done so organically.

In both tactics you are giving people the chance to tie their world view in with the results and letting them have a stake in the outcome. You are welcoming that hammer they wield but helping them see that there are many different nails to hit.

Keep in mind however that you are just as guilty as they are. Spend too much time in the world of optimization and you will start to feel like no one has any idea what they are doing and that all ideas are going to fail. It is even more important for you to challenge yourself and for you to go beyond your comfort zone in where you let testing going. Make sure you include ideas from others as much as possible, even if you are sure they are not going to work. Make sure you tie optimization in on actions that you feel might not comfortable or worth your time. Remember that the smarter someone is, the more likely they are to be impacted by biases and that you serve no good to the organization or yourself if you are not more vigilant against your own biases then you are against others.