Stop Experimenting. Start Deciding

It’s important to be data-driven, but teams have to be careful that it doesn’t become a crutch. Here’s my take on why simply protecting your KPIs might be holding you back

We loooove to say we’re data-driven. We run experiments. We track metrics. We celebrate learning velocity like it’s a badge of honour. But we must be careful that our strive for insights and metrics (read: perfection) doesn’t shackle us, preventing us from making smart decisions based on good product sense and taste.

Image of a woman holding a pen and notepad. She looks bewildered as numbers, equations, and question marks float around her head

Speaking of taste…

In a conversation with Lenny Rachitsky, Shreyas Doshi (ex-Stripe, Twitter, Google, Yahoo) says that “Taste is the ability to identify what is really good without needing to see its results” (watch the full episode here). It requires zero taste to say, now, that touchscreen phones are better than phones with a physical keyboard. But do you remember that when Apple launched the touchscreen iPhone back in 2007, critics mocked the lack of a physical keyboard, saying it would be slower and impractical compared to a Blackberry, or a Nokia. Do you remember Blackberries and Nokia phones, I wonder?

Similarly, one would say that it’s pretty obvious that short form content is a super engaging format, especially for young people, because all you need to do is look at the companies who are leveraging this to drive engagement among their users to know that this is a valuable offering. It would have looked risky at the time for companies who first took a bet on this back in 2013/14 (short attention spans, mobile-first, user-generated loops?), but it reflected deep intuition about shifting media consumption.

Taste is the ability to identify what is really good without needing to see its results.
Shreyas Doshi

Anyway, I digress. Product sense and having good taste is a topic for another article. 😀

Back to experimentation. I’ve seen in organisations where this becomes a crutch, and a way for teams and/or leaders to avoid making a call. Every team needs to get to the point where they so deeply understand their users, and so deeply understand the competitive space in which they operate, that data is simply directional. It tests a hypothesis enough to de-risk a feature, allowing the team to move forward more confidently.

So instead of asking for more and more tests, leadership is able to Just Decide Already (JDA, you read it here first!)

The comfort of endless testing

Experimentation gives us safety. It signals that a team has taken steps to de-risk a feature or proposition. It makes us feel like we’re doing the “right” thing: staying curious, not just jumping to assumptions.

But too often, it becomes a holding pattern. We keep testing because we’re afraid to be wrong. Because deciding means owning the outcome. Certainly, in places where leaders are quick to blame the team/blame you as the product manager if something goes wrong, I can empathise why you would feel it’s ‘safer’ to completely rely on an experiment effectively telling you what to do.

Truth be told, experiments only help you reduce uncertainty, not remove it completely. The rest is product sense + gut instinct + knowing your users and market + having the courage to just go for it!

An image of a woman looking stressed. She is shouting at her laptop ‘Experiment, oh experiment on the wall. Just tell me when I should launch’. A message on the laptop screen reads ‘That’s not my job’.

The trap of “Do No Harm” experiments

Ah, a ‘do-not-harm’ experiment. This is when leadership wants you to validate a new feature by showing that it doesn’t harm the existing, primary metric. Suddenly, your experiment isn’t truly an experiment anymore. It’s a safety exercise dressed up as ‘science’. Unfortunately – in my view – this is how innovation quietly dies in an organisation.

I get it. No one wants to break something that’s working perfectly well, but that’s how organisations get stuck in doing ‘ok’ versus truly innovating. The best innovations always cannibalise something in the short-run. Take a look at these two examples.

When Apple (I promise, they aren’t sponsoring this post) launched the iPhone, it could have excluded music playback as one of the features in order to protect revenue from iPod sales. And that would have made sense, right? Not Apple. They deliberately integrated a music player with a phone, essentially making their flagship product at the time redundant. They made it obsolete in order to bet on a bigger future. And what a bet it was. In the last 3 months of 2007, iPods made up 42% of Apple total revenue – a sweet $4 billion (ref). Fast forward to today, iPhone sales made up 51% of their total revenue in 2024, around $195 billion (ref).

The same story is seen with Henry Ford (one of our favourite product examples), when he built affordable cars. He didn’t optimise for faster horses, or worry about cannibalising weekly horse usage (WHU? Just go with it 😀). Instead, he completely changed how people travelled.

AI generated image. Shows Henry Ford and a horse with wheels instead of hooves. Text reads: If I only looked at the data, I would have given people a faster horse.

Now, I’m not suggesting being reckless, and I imagine whoever signed off that decision at Apple would have been pretty nervous at that time. But sometimes you have to trade a little dip in the short term for potentially larger gains in the long term.

It’s about defining what “acceptable harm” looks like, setting smart guardrails, and knowing which metrics truly matter in the long run.

When the numbers say “no” but your gut says “yes”

So I know I said I would cover product sense in another article, but just quickly...

Not every decision will come neatly packaged up in a dashboard, with flashing red arrows saying “launch this feature on the 19^th of November at exactly 11:42am”. Sometimes the quantitative data might look flat, even inconclusive. However, the qualitative signals will be telling a different story.

You’re able to read between the lines when you speak to users.
You can see it in the comments they leave on your App Store reviews (or your competitors...)
You can hear it in the subtleties: “I love this in <insert competitor’s product name here> but I didn’t expect you/your brand were cool enough to have this feature too!” Ouch

Those are the moments when your instinct / product sense matter more than numbers and data.

The best product managers know how to evaluate both – evidence plus instinct. They’ve been building products long enough to know how to feel what the data can’t yet prove. They know when they’ve reduced as much uncertainty as possible that they can make the necessary decision with enough confidence. They also know that if they wait until they have 100% certainty, they will lose time and time again to their competitors.

The real lesson

Data gives you direction, and experiments give you confidence. But ultimately, at some point you’ll need to put your big girl/boy pants on and make a decision. This is what gives your team momentum. This is what sets you apart from the rest.

Besides, what’s the worst that can happen?