A Defence of Product Intuition
Is a hyper focus on data-driven decisions harming your product development?
There’s still something to be said for good ol’ product intuition. Data and product metrics have become more accessible and normalised in the last decade. With this, and an improved understanding of product best practice, product decisions based on data have become the norm. This is great, and essential to building good products well. However, I would argue that a hyper-focus on data can create worse products. Let’s talk about some of the scenarios where an over-reliance on data can be harmful and how you can find that sweet spot.
Pre-product start-ups
In a pre-product start-up, how do you balance between research, experimentation, and building a thing and getting it live? You are going to have very limited quantitative data. This is understandable, you haven't launched a product yet and so don't have access to any product metrics. You become dependent on third party research, surveys and user interviews to understand the most important problems that need to be solved. This is vital.
But, once you know the problem to solve that people will be willing to pay for, what then? There are hundreds of ways you could solve a thing, especially in a greenfield or innovative space. If you need to test every single one in depth before you build a thing, you either have an endless runway, or you’re going to be going back to your investors with nothing to show but some PPT slides and a request for more money.
It's at this stage you need to have a vision for what will make your company and product successful. Instead of relying on the masses, you need to rely on the experts or on your team. Based on their experience building products, working in the domain, or understanding the market, what does the team think will be the most valuable thing to build first? Use some quick and low lift techniques to get fast and imperfect data e.g. surveys or a few generative concept tests. This should highlight any major red flags but prioritise the rest of your effort on building the quickest version that is valuable and getting it to market. After this point, you have a live product and can start getting the data you need alongside your continuous research. It has the bonus of giving you a revenue stream which is what you need early on if you want to test, experiment and build until you’ve fully achieved product market fit.
Large Organisations with too much data
At the opposite end of the scale are organisations that have so much data they haven’t quite figured out what to do with it. Quantitative data tells you what is happening (e.g. CTRs, NPS, AARRRs) but you don’t know why something is happening. And because stakeholders have competing opinions and interests in an organisation, data will be twisted to justify competing viewpoints. A few examples: we have an acquisition problem, so let’s double our marketing budget before asking if we have a valuable product offering; people widely reference experiments that haven’t achieved statistical significance; or there are frequent examples of senior stakeholders mistaking correlation for causation.
The answer is not to stop using data. It’s being more intelligent in how you leverage that data. First, pair qualitative data with quantitative data, make sure you are building a deep and personal understanding of your users and their lives, and make sure you take the time to understand market and industry trends. This will feed your product intuition, your spidey- senses should tingle when you are mistaking correlation for causation. Second, use data in a targeted way. Take the example above: there is a low acquisition rate, but the rest of your pirate metrics look to be inline or better than market. Marketing efforts are one of many potential answers, is your product too niche, are you going after the right user segment etc? Go back to basics before jumping to a conclusion.
The other anti-pattern in this environment is when corporate caution meets data driven decision making. All product decisions have to be supported by data. To get access to data you need to have data about why you need that data, and you end up in a death loop. There’s a healthy balance that can be found, one in which processes protect user data while also allowing data to be used for problem exploration by PMs. Tweak your processes until PMs have the data they need, and tweak your culture until PMs can build small, low-risk things that have high confidence without another large experiment that will then generate data they need to proceed.
All hail the North Star Metric
Okay, focus is super important. We know that. Trying to be all things to all people is why a lot of products fail. They lose their focus on the value they need to unlock for their core users. However, product is nuanced, and so it needs nuanced measurement. If your whole focus is retention for example, and the figure the organisation checks on and builds its plans around is your customer churn rate, you are going to encourage your people to put in place short term measures to increase this figure e.g. incentives, or features that only serve the existing user base. Growth will be neglected, and your product will be boxed out by faster growing competitors. If, instead of shaping your strategy around a metric, you base it around the value you want to add to the world, you will have a longer term more nuanced roadmap that allows for short term dips in key metrics for bigger long-term gain.
Conclusion
Data is important for being able to do product management well, but if quantitative data is your sole or top driver for making product decisions you aren’t going to make the best product you could. Care about the qualitative stuff as much as the quantitative stuff. Invest in data literacy and scrutinise your data with product intuition, don’t just take it at face value. Care about your product’s purpose in the world, don’t just care about one figure going up or down in the short term. Data, like anything else, works best when paired with human intuition.