Everyone says that digital transformation is hard, and that it is messy. It’s a truth that is acknowledged, but only kind of.

Companies seeking to fundamentally change how they operate to remain competitive amid the rapid advances in technology can usually accept that they are behind the curve. However, they have to envision digital transformation as a straight line to where they need to go. Employees need to back the initiative and do the groundwork to make it happen. Sponsors (and for public companies, shareholders) need to fund it.

There may be talk of leapfrogging, of taking advantage of best practice established by others as a shortcut to business results. Perhaps there will even be some acknowledgement that iteration and experimentation will be needed. This is usually imagined as a gentle process of optimising through the new suite of testing tools that the team will shortly have at their disposal.

What’s harder to accept is the scale of the unknowns, and the impossibility of anticipating everything perfectly. If those first few steps towards transformation falter, and the business results don’t immediately materialise, the path to this hopeful future suddenly doesn’t look so clear. The pressure ratchets up. How can the first launch be floundering, when we thought it was a sure-fire winner? Wasn’t this supposed to solve all our problems, not create new ones?

This is a key moment, and one that can either strengthen a business’s commitment to transformation or irrevocably lose its confidence. If you can’t address the issues quickly, expect the questions to become more pointed: Why are we even doing this? Were we actually better off before?

If it becomes clear that the approach that we started out with actually isn’t going to deliver the outcomes we need, we need to adjust while keeping teams and sponsors on side. Acknowledging when we have got something wrong makes us feel vulnerable, particularly when things aren’t going well. However, it is only by recognising the gap between what we thought we knew and what have since come to know that we become empowered to act from a place of deeper knowledge.

Can we avoid being wrong?

We want to believe, as we do our upfront analysis, that we have mapped all of the risks, issues, and mitigations, and that there are no lurking surprises. Unfortunately, while being thorough is essential, it isn’t always enough.

Digital transformation generally takes place in larger, established companies. That means a complex environment, often with legacy systems, data of variable quality that has accumulated in multiple places over the years, and different teams working with limited visibility of each other. It may be a real challenge to get even an accurate view of customer numbers or conversion rates because data isn’t joined up enough. You may hear dozens of conflicting things about a how a certain system or process works, or find that the tacit, unspoken knowledge in people’s heads is entirely at odds with what has been documented. You may surface some of this before you launch a new product, but if you don’t foresee everything, the launch still may not go as planned.

Here are some potential gotchas:

• When offline processes are digitised or automated, it can expose a gap between what the business believes is happening and actual day-to-day operations. If a lack of psychological safety prevents front-line teams from being fully honest during the design and implementation, you will be building for a reality that doesn’t exist.

• Legacy data (e.g. duplicate records, missing fields) may cause unexpected outcomes if this isn’t surfaced during the testing process. There may be a lot of scenarios that require special handling that nobody was aware of, and things that were considered “edge cases” actually might make up a significant proportion of your customer base.

• Users may half-remember old journeys, and this can affect how they perceive what you build, for example if customers accessed services using personal details but didn’t previously need to set a password, they may not actually realise that they don’t have an account when you roll out a shiny new login. Your design may need to factor in these different mental models.

Why being wrong feels wrong

As product managers, it is our job to surface this kind of misalignment as quickly and as cheaply as possible: by talking to users, by testing prototypes, by releasing product increments that are well-enough realised to prove or disprove the concept driving them while keeping the scope of development as small as possible.

When an unknown unknown surfaces, it feels awkward. People will ask, “Why are we only finding out about this now?” This is a fair question. It’s worth exploring whether, with hindsight, you could have got this information any other way. Be honest with yourself. Unpack what you should have done differently and ensure that you build that into how your team approaches problems in the future.

You may conclude that there was no way you could have foreseen how some got this information more efficiently. Indeed, the feedback loops put in place to monitor new work might suddenly shine a light on something that has been failing silently, perhaps for years. However, in the face of counterintuitive results, the expectation of what “should” be happening can distort perceptions of what is happening. People may then discount mounting evidence, saying things like, “That can’t be on fire or we’d have heard about it” even as call numbers rise or conversion rates fall. It’s important to be open to new information and new interpretations of what we think we already know, however uncomfortable that might be.

Can you prepare to be wrong?

As we saw with the public health messaging during Covid, a narrative that adapts as more information emerges is more difficult to land than one that presents itself as unchanging and authoritative. Changing tack during a digital transformation can be similarly challenging, but when communicated well, it can ultimately reinforce the benefits of transformation: a focus on business value, the flexibility to adapt, empowerment for teams to act swiftly on new information. If you build this into how you structure your communications at the start of your transformation, you can be prepared if you do need to pivot your approach.

Having an over-arching vision of the experience you will be enabling through your transformation efforts set out from the start is critical to keeping everyone focussed when so much else is in motion. Keep reiterating the vision; it will become more powerful through repetition. The answer to “Why are we doing this?” should be clear not only to you, but the entire team. If you need to rethink how you will achieve that vision in light of new information, this becomes an easier story to tell because you are countering uncertainty with familiarity and clarity of purpose.

When it comes to delivery, I recommend prioritizing unblocking the biggest challenges that are standing in the way of creating the experience you want to create: maybe this is getting an accurate conversion rate across multiple systems, or managing a handoff between platforms that currently feels really awkward to the end user, or aligning multiple channels to a single source of truth, or something else again. When teams come under fire to deliver something quickly, it can be tempting to continue polishing journeys based on existing workarounds, but ultimately these will never be able to deliver the step change that the business demands. Releasing a “thin slice” that delivers an end-to-end experience that is pared back but still meaningfully better than what users had before will take you in the right direction, faster. It might not be an immediate success, but that is exactly why you do a thin slice and not a fully realised solution.

How do you move past being wrong?

If you have a difficult release, tell that story, not as a failure, but as a turning point in your digital transformation. Surface what you have learned and show how it is helping you to realise the business’s ultimate goals by demonstrating whether you have a viable way to address its biggest challenge. Finding out that a particular solution isn’t going to work is still valuable. If you do have a viable solution but there are smaller issues that are creating a lot of noise, how quickly can you address these? Ensure that the things that are already better are recognised, as these may be taken for granted or not noticed at all.

It’s still more challenging when it’s not obvious why you’re getting a certain outcome, but start with what you know and dig into that. Is your data accurate? Can you drill down further to see where people are struggling, or dropping off? Can you get closer to your customers or front-line teams, to really understand what’s going on? While it can be frustrating to be confronted with data that doesn’t appear to make sense, going beyond received wisdom is how the most critical insights happen, the ones that will enable real progress. These are what I think of as the uncomfortable epiphanies.

While it’s not always possible to predict when inspiration will strike, you can make breakthroughs more likely by creating an environment that makes it safe to share potentially unflattering information or challenge existing narratives. Leading by example and being open with your stakeholders about what you don’t know and how what you do know has changed over time requires courage, clarity of communication, and confidence in where you are headed to give you enough credibility for your message to land. Embracing the idea that you might need to flip what you think you know on its head unlocks the possibility of true transformation, not only by giving you the space to adapt when new information emerges, but also by fostering a culture that is open to insight and change.

Digital transformation is not an easy process because the technical side of it is only a fraction of the change. By acknowledging that challenge up front and setting the scene for the potential need for recalibration, you will be ready to receive and quickly act if you do happen to have an uncomfortable epiphany. Your new insight will propel you forwards into the now slightly less unknown.

TLDR

Most enterprise data platforms are built like traditional infrastructure: costly, slow, and delivering value too late.

A better approach is to treat data platforms as products.

By identifying high-value domains, building thin end-to-end slices, and releasing data services incrementally, organisations can unlock value in weeks rather than years.

Introduction

You’ve probably heard expressions like “single source of truth”, “enterprise data model”, “foundation phase”, or “three-year transformation programme”. They usually come with large budgets, slow delivery, and teams waiting years to see any value.

If this sounds familiar, this article may offer a fresh perspective to help your organisation apply product principles and agile thinking to quickly drive data value.

I first worked on data products when running analytics for a startup telecoms company in Cambridge in the late 1990s. The business’s needs drove the architecture, not the other way around. We had visibility of live systems and access to 24 hours of queryable operational data. At the time, this was unusual.

Ten years later, I rebuilt a similar capability at American Express, combining live transactions with management information.

All of this is to say that, in my experience, it is highly likely that the systems that are the source of your data are rarely what you need now, and when you get what you need, it’ll be too late.

The real issue is not that your company lacks a data platform, but that most are built as infrastructure projects rather than products.

What does this do to your organisation?

In developing new AI products for clients over the past two years, I’ve repeatedly encountered issues with their data, both customer interaction and product data, that forced us to work around their limitations. The problem usually isn’t a lack of data, but rather inaccessible formats in unavailable systems or data only in employees’ heads, any of which takes time to extract.

Data’s importance is rising fast, yet most platforms and approaches are too slow to keep up. This gap puts your organisation at a disadvantage.

Understanding why organisations struggle with data requires looking at what’s changed.

To be honest, some of my colleagues and I got frustrated because we didn’t see why the status quo, getting slightly better, a bit late, should stand. Why should we wait for the data we need?

So we set about asking our data experts the hard questions. Questions about what an MVP data service might look like, what tools you might use, and what you could do without. What are the limitations and risks of doing this? What are the benefits and opportunities?

We took a minimal viable product (MVP) approach to build data capabilities, reviewing which tools allowed rapid iteration and which were limiting, just as we aim for a short, efficient CI/CD pipeline in app development.

What changed isn’t just technology. The real shift is applying product thinking to data platforms.

We decided to show, through practical action, how building a Product-Focused Data Platform can give our organisation a real competitive advantage.

With this product-led perspective in mind, how can you put it into practice?

I won’t reveal our specific solution, because your organisation may require different tools, and sharing details could compromise data security. Instead, I’ll outline the principles and steps you should consider.

1. Learn the language of data platforms

Data platforms have a unique vocabulary: Lambdas, Pipelines, heavy DevOps, and Gold, Silver, Bronze layers. Ingestions, services, event streams, APIs. Talk with your data teams, learn the terms, and stay up to date in this fast-changing field.

2. Create a simple operating model

Model how you’d ingest, process, and supply data, keeping the language accessible for all your colleagues. Visuals are often just complicated architecture diagrams, too complex, and seen as someone else’s expertise. All organisations rely on data, so make your model clear and accessible. Ironically, poor communication leads to lost battles over data.

3 - Take a step back

Proceed by considering which data matters to your organisation and why. You need the big picture. We talk about zooming out, going big, and thinking about what matters now and for the future. Think beyond how you do things today. At Amex, what really mattered was integrity. In a business built on reputation, customers expected us to have integrity. This meant transaction data and customer data were most important. What, one or two things, really matter in your organisation if you zoom out?

4- Domains

Now zoom in on your data domains. Understand how your big-picture view breaks into smaller components. Don’t be constrained by current operations; focus on what matters over the long term. For my team, customers—split by details, context, and behaviour tops our list.

To identify ours, we asked what core was, what mattered, and what was a commodity. This process helps you detail domains and anticipate what may change as your organisation evolves.

5 - Services

Now convert this knowledge into clear services. These are often APIs or Event Streams, depending on needs. APIs answer questions; Event Streams announce events. Choose what delivers the most value to your users. Simple services bundled together create real value. Plan upgrades and ensure most teams’ needs are met most of the time.

6 - Value

Next, identify where your data creates the most value. Since you likely have existing data flows, ask what gaps you can fill now to add value and get momentum.

Building a better way for your organisation to access data is key. We use a simple approach—evaluating value as Reach × Impact—but choose what works for you. Focus on surfacing high-value opportunities.

7. Deliver thin end-to-end slices

Slices in the data world are pieces that ingest, process and supply data to teams that use it to make a difference - value delivered.

To maximise value, build thin, functional slices that teams are going to use now. Once a slice works and delivers value, repeat with more and don’t forget others can use your previously created API’s. This approach accelerates results, uncovers problems early, and limits risk and management overhead.

8 - Build Just Enough

Organisations often try to build the “perfect” data platform before anyone uses it.

Instead, build something usable but minimal: a service that runs weekdays, handles non-critical data, and demonstrates value quickly.

Once teams depend on it, you can harden reliability and scale.

9 - But Stay ahead of the org

Keeping ahead requires both tactical and strategic actions.

Tactically, as more people use your API, it shifts from useful to indispensable. Ensure service quality keeps up, move from 9-5 support to 24x7, from downtime to high availability. The MVP approach grants incremental releases while earning and preserving trust by being ahead of the curve.

Strategically, you’ll want to look ahead to business changes and ensure your model accommodates them. Work closely with the board to get an early view, and your platform will continue to support the organisation rather than become a dead weight.

Building a Product-Led Data Platform is about mindset, not just technology.

Treat data services like products. Deliver value in slices. Stay close to leaders and the teams that use the data. By zeroing in on these takeaways, you can achieve results faster with greater impact and maintain this cadence.

Now it’s your turn to apply these principles and share your story.

I want to hear from you in the comments about your experience with data at your organisation. Can you get the data you need to deliver value, or are you held back? And if there are limitations, where do they come from? Lastly, what are you doing to turn this around and help create a Product-Led Data Platform now?

In previous articles, I explored how systems thinking techniques such as Soft Systems Methodology (SSM), System Dynamics (SD), and Critical Systems Heuristics (CSH) can help product managers understand complexity.

Related articles:

• The basic principles of Systems Thinking and how it can be used in Product Management.

• How applying Soft Systems Methodology (SSM) to can help you understand complex socially rooted problems (Peter Checkland).

• How applying Systems Dynamics (SD) can help you understand complex situations and causality (Jay W. Forrester).

• How applying Critical Systems Heuristics (CSH) to Product Management can help identify assumptions, boundaries and power dynamics (Werner Ulrich).

These approaches are powerful, but they focus primarily on understanding systems, but they do not tell us whether the system itself is capable of functioning.

Viable System Model

The Viable System Model (VSM), developed by Stafford Beer in the 1970s, addresses that gap. Rather than improving processes, VSM asks a more fundamental question:

What must exist for a system to remain viable?

Beer argued that any viable system, whether an organisation, service or product ecosystem must contain five interacting functions:

• System 1: Operations - The parts that deliver value (teams, services, products)

• System 2: Coordination - Mechanisms that prevent instability between operational units

• System 3: Control - Internal governance, optimisation and resource allocation

• System 4: Intelligence - The ability to sense and respond to the external environment

• System 5: Policy - Purpose, identity and ultimate decision-making authority

These are not organisational layers. They are functions that must exist somewhere within the system. When they are missing, weak, or disconnected, the system may continue to operate but it will not produce coherent outcomes.

Applying VSM

VSM is most useful as a diagnostic and design lens, rather than a rigid method. Its value in Product Management comes from helping us ask whether the system around a product, service or programme is actually capable of functioning coherently over time. A practical way to apply it is to work through the following steps.

Step 1: Define the system of interest

Start by deciding what system you are actually analysing. This matters more than it first appears. If the boundary is too narrow, important causes of failure will sit outside it. If it is too broad, the exercise becomes too vague to act on.

The system of interest might be:

• a single product area

• a service end to end

• a programme spanning multiple teams

• a wider operating model around a product

The key question is:

What is the smallest meaningful system that must function well in order for the intended outcome to be achieved?

For example, if a team owns only the frontend journey but the real problem depends on policy, operations or supplier constraints, then the actual system is larger than the product team itself.

Step 2: Clarify the system purpose

Before mapping structure, define what the system is there to do. This sounds obvious, but many product environments contain multiple competing versions of success:

• delivery to plan

• user value

• policy compliance

• operational efficiency

• commercial return

VSM works best when you ask:

• What is the core purpose of this system?

• What outcome is it supposed to produce?

• For whom?

• Over what time horizon?

If there is no shared answer, that is already a structural signal. It often points to weakness in System 5: Policy, where purpose and identity should sit.

Step 3: Identify the operational units

Next, identify the parts of the system that actually do the work.

These are System 1: Operations. In product contexts, these might include:

• delivery teams

• service teams

• operational teams

• customer support functions

• supplier teams

• platform teams

• regional or business units

The important thing is not your formal org chart, but the units that genuinely carry out value-producing activity.

Ask:

• Where is value created or delivered?

• Which teams or units are operationally distinct?

• Which parts of the system need to function well day to day?

At this stage, many people discover that the system is more distributed than they assumed.

Step 4: Map coordination mechanisms

Once you know the operational units, look at how they are coordinated.

This is System 2: Coordination. Its role is to prevent instability, duplication, conflict and fragmentation between operational units.

In product environments, coordination mechanisms might include:

• ceremonies across teams

• shared standards

• common definitions of done

• design systems

• architecture principles

• roadmap alignment

• dependency management

• operating rhythms

Ask:

• How do these units avoid working against one another?

• Where are dependencies managed?

• What keeps the overall system stable?

If the answer is mostly informal heroics, escalation, or “people just talk”, coordination is probably weak.

Step 5: Identify control and governance

Now look at how the system allocates resources, monitors performance and makes internal decisions.

This is System 3: Control. It includes the mechanisms that optimise the system internally and ensure operational units are functioning as expected.

In product settings, this may include:

• portfolio governance

• leadership forums

• budgeting

• performance review processes

• prioritisation decisions

• risk and assurance

• delivery oversight

Ask:

• Who decides what gets funded, prioritised or stopped?

• How is performance monitored?

• Where does operational accountability sit?

• What happens when teams drift off course?

This step often reveals imbalance. Many organisations have very strong System 3 functions and relatively weak coordination or intelligence.

Step 6: Identify intelligence and adaptation

Then examine how the system senses external change and adapts.

This is System 4: Intelligence. It is responsible for looking outward and forward.

In product terms, this could include:

• user research

• market analysis

• policy scanning

• technology strategy

• service performance trends

• horizon scanning

• experimentation and learning

Ask:

• Who is looking beyond current delivery?

• How does the system understand user, market, policy or environmental change?

• How are future risks and opportunities brought into decisions?

A common pattern is that organisations gather research or insight, but it is disconnected from actual decision-making. That means System 4 exists in fragments, but is not functioning effectively.

Step 7: Identify policy, purpose and authority

Next, identify where the system’s overall identity and direction are defined.

This is System 5: Policy. It is concerned with purpose, values, strategic coherence and ultimate authority.

In practice, ask:

• Who decides what this system is ultimately for?

• Who resolves tensions between short-term delivery and long-term direction?

• Where is the final authority on trade-offs?

• Is there a coherent identity or just competing demands?

If no one can clearly answer these questions, the system may be operationally active but strategically incoherent.

Step 8: Look for gaps, overloads and distortions

Once the five functions have been mapped, look at where the system is structurally weak. Common patterns include:

• strong operations but weak coordination

• heavy governance but poor intelligence

• unclear purpose with competing authorities

• insight gathered but not acted on

• responsibilities split across boundaries with no one holding the whole

This is the point where VSM becomes especially useful. Instead of saying “the team is struggling”, you can say something more precise, such as:

• coordination between operational units is weak

• policy is unclear

• control dominates intelligence

• the system boundary excludes a critical dependency

That shifts the conversation from blame to structure.

Step 9: Test the system boundary

This is one of the most important steps. Once you have mapped the system, ask whether the boundary you drew is actually valid.

For example:

• Are key decisions sitting outside the system?

• Are important operational dependencies excluded?

• Is the team being held accountable for outcomes it cannot influence?

• Is policy or governance outside the analysed boundary even though it shapes performance?

If so, redraw the boundary and test again. This is where VSM becomes particularly useful for Product Managers in complex environments. It helps reveal when the “product system” is actually inseparable from policy, operations, supplier relationships or service design.

Step 10: Use the findings to redesign, not just diagnose

The final step is to decide what needs to change. This is where the exercise becomes practical. The goal is not just to produce a map, but to identify structural interventions such as:

• introducing clearer cross-team coordination

• strengthening user or market intelligence

• clarifying decision rights

• redefining the scope of the product system

• reducing overbearing governance

• aligning policy and delivery around a shared outcome

The question at the end is:

What changes would make this system more viable, not just more efficient?

That distinction matters, as a system can be highly efficient at producing the wrong outcomes.

A simple workshop approach

If applying this with a team, a straightforward structure is:

1. Define the system of interest.

2. Agree the system purpose.

3. List the operational units.

4. Map current coordination, control, intelligence and policy functions.

5. Identify missing or weak functions.

6. Test whether the system boundary is correct.

7. Prioritise structural improvements.

This can be done on a whiteboard or in Miro without needing to use all of the formal VSM language. In many cases, the concepts are more useful than the terminology.

Example: When delivery works but the system doesn’t

In a large public sector programme, multiple teams were delivering against a shared objective. Each team:

• maintained a backlog

• delivered regularly

• reported progress through structured governance

From a delivery perspective, the programme appeared healthy, work was moving, milestones were being met and activity was visible. But when asked a simple question:

“What has improved as a result of this work?”

The answer was unclear. Outcomes were inconsistent and progress towards the overarching goal was difficult to demonstrate.

Initial diagnosis: a delivery problem

The initial assumption was that delivery needed improvement. The focus turned to:

• refining backlog structure

• improving prioritisation

• increasing reporting clarity

• strengthening delivery discipline

These interventions improved visibility and consistency, but they did not change the underlying outcome.

Applying a VSM lens

Instead of continuing to optimise delivery, the programme was examined using the Viable System Model. The system of interest was defined as:

the set of teams, governance and supporting functions responsible for delivering the programme outcome

Each of the five VSM functions was then assessed.

System 1: Operations (present and functioning)

Delivery teams were active and capable.

• work was being delivered

• teams were structured

• outputs were produced consistently

There was no fundamental issue with execution at a team level.

System 2: Coordination (weak)

Coordination between teams was limited.

• dependencies were managed reactively

• teams worked to local priorities

• duplication and gaps appeared across workstreams

There was no strong mechanism ensuring the system behaved as a coherent whole.

System 3: Control (strong)

Governance and oversight were well established.

• reporting structures were clear

• progress was monitored closely

• escalation routes existed

However, this control function focused on:

• activity

• delivery status

• adherence to plan

rather than:

• system effectiveness

• outcome alignment

System 4: Intelligence (fragmented)

Insight into the external environment existed, but was not integrated.

• user research was conducted inconsistently

• policy and regulatory context evolved independently of delivery

• learning was not systematically fed back into prioritisation

As a result, the system struggled to adapt.

System 5: Policy (unclear)

There was no single, shared definition of success. Different parts of the programme optimised for different outcomes:

• delivery to plan

• compliance

• stakeholder expectations

• local team objectives

This created a lack of coherence across the system.

The key realisation

At this point, the issue became clear. The problem was not that teams were failing to deliver. It was that:

the system they were operating within was not structurally capable of producing coherent outcomes

What changed as a result

The response shifted away from delivery optimisation towards structural change. Three key interventions followed.

1. Strengthening coordination (System 2)

• Introduced cross-team planning and dependency mapping

• Established shared artefacts and alignment points

• Reduced duplication and fragmentation

2. Reconnecting intelligence to delivery (System 4 → System 3)

• Regularly integrated user, policy and performance insight into prioritisation

• Ensured external context influenced internal decisions

• Shifted focus from activity to relevance

3. Clarifying purpose and decision authority (System 5)

• Defined a shared outcome for the programme

• Established clearer ownership of trade-offs

• Aligned teams around a common direction

The result

The volume of delivery did not increase significantly. But:

• work became more coherent

• decisions became more consistent

• outcomes became clearer and easier to demonstrate

The system had not been made more efficient, it had been made more viable.

Why this matters

Without applying a structural lens, the programme would likely have continued:

• refining processes

• improving reporting

• increasing delivery discipline

All of which would have made the system appear healthier, while leaving the underlying issue unresolved.

What this means for product management

The example above is not unusual.

Across product organisations, a similar pattern emerges. Delivery is active, governance is visible and yet outcomes remain inconsistent. The instinctive response is to improve execution. E.g. Refine prioritisation, restructure backlogs, introduce clearer processes. But this assumes the system itself is capable of producing the intended outcome.

In many cases, it is not.

What appears as a delivery problem is often structural. Teams can be effective, disciplined and productive, while operating within a system that cannot produce coherent results. Improving execution in this context increases activity, not impact.

A key part of this is coordination. In many organisations, coordination is not designed, it is assumed. Teams are expected to align, but dependencies are managed reactively, duplication and gaps emerge and coherence depends on individual effort rather than system structure. Without deliberate coordination, the system behaves as a collection of parts rather than a whole.

At the same time, control is often overrepresented. Governance, reporting and oversight are well established because they are easy to formalise. They provide visibility and reassurance. But they tend to focus on internal activity rather than external relevance. Meanwhile, intelligence, understanding users, policy context, or environmental change is fragmented or disconnected from decision-making. The result is a system that is well controlled, but poorly adapted.

The boundary of the system plays a central role in this. It defines what can be influenced, what can be improved and what remains out of reach. In many product environments, teams are held accountable for outcomes that depend on elements outside their defined system. Policy, operations, suppliers or governance sit beyond the boundary, yet shape the result. This is not a failure of delivery. It is a failure of system definition.

This is why improving efficiency alone is insufficient. Techniques such as prioritisation, roadmap refinement or value stream mapping can improve flow within a system, but they do not make an unviable system viable. If key functions are missing or misaligned, increasing efficiency will simply produce more output, more effort and the same outcomes.

This points to a different understanding of Product Management in complex environments. The role is not only to optimise delivery, but to define and shape the system itself. That means asking what the system includes, how it functions, where its boundaries sit, and whether it is capable of achieving its purpose.

Without that, product management risks becoming well-executed delivery within an unworkable system. And if the system is not viable, improving delivery will only produce more of the same.

Systems Thinking for Product Managers (to be continued..)

This is the fourth article in a series I’ll release within Product Breaks, focusing on Systems Thinking methodologies and Product Management. Further articles will aim to cover the topics:

• Strategic Options Development and Analysis (SODA)

• Hard Systems Thinking (HSM)

• Systems Failures Method (SFM)

Good team culture leads to successful outcomes – this is one of those obvious facts that everybody ‘knows’, but in my experience, culture and ways of working are the first things to be left behind when pressure is placed on a team to get something done.

As the industry develops and more companies adopt operating models orientated around product, we’re also seeing less of those roles that had team culture as a dedicated focus (think scrum master and delivery managers), which means this crucial aspect of successful delivery has become a shared responsibility, and is in danger of being forgotten.

Product managers can make the difference - but how do we make our teams a great place to be, while also chasing ways of working that actually get us better product outcomes?

I’ve been reflecting on five practical methods that help you do both.

1. Democratise decision-making wherever possible

Product is often called in as the ‘decision maker’ where prioritisation is needed, and obviously product set (and change) the overall strategy that the team is chasing. Even given that, I would say that the actual decisions that have to be made solely by product are lower than you think, and they’re usually tied to that handful that include risk or uncertainty.

This means there are many, many product decisions that can be made most effectively in partnership with the whole team. If you prioritise the shared forums and space to do so (e.g. workshops, ideation sessions), you increase your team’s investment in those decisions, and raise individual’s confidence in their understanding and judgement of the product. Here’s a test – ask somebody in your team why you’re working on the current priority. If they can’t give a good answer, maybe you didn’t invite them into that decision. Maybe you didn’t let them decide anything!

2. Normalise sharing WIP

There’s nothing more disconcerting than a product strategy that’s half finished – perhaps because we ultimately look to roadmaps, plans, strategies, etc., to give us certainty about where to go. I think there’s a particular pressure for product managers (perhaps down to a drive to ‘justify’ the work we do) that we need to present plans at the point where we’re most confident in them.

But if you share half-baked, hole-ridden, unfinished plans and invite everybody to give a first response, you’re going to instantly engender a culture of radical candour and feedback. You need to show vulnerability for others to do the same – and ultimately only in a culture of vulnerability and trust can decisions be fully scrutinised on their own merits without ego.

3. Make sure your team rhythms zoom out as well as in

We’re all clear on weekly ceremonies and their purpose in getting us to reflect, plan and communicate. But they’re very detail focussed. What about the ceremonies where we invite our team to look at the one year, or even the three year plan? There’s often an assumption that this is relayed by others and the team don’t join in at that level, but this misses out on the excellent opportunity to strategise upwards.

Ideally, whenever strategy ‘comes down’ in the organisation, you want to be in a position to push strategy upwards too, this is where top level thinking meets team level insights. So why not actually bake this into your rhythm? I suggest that at the least you have a quarterly session where you look to deliberately set the lens broader than your specific remit and invite your team to be creative with it.

4. Focus your energy on resources other people can use

This is an interesting one – but it’s a reflection of a shift I’ve seen when working with more empowered teams. There is always a bunch of time and effort required for effective stakeholder communication. Decks are put together, narratives are nailed. These comms tend to be for a specific context – and although they’re high effort and ‘urgent’, they tend to be one-use.

I’ve found it much better to focus on evergreen strategy docs that have a wider intended audience, with the aim of team members using them in their day to day comms, whether that’s with other teams or to frame thinking/deep dives. A great slide on ‘why this problem matters now’ will not only be extremely useful to have on hand, but could empower a particular team member to communicate that concept well in a meeting that only they attend, perhaps for a specific discipline. That’s going to work wonders for your consistency of messaging.

5. Prioritise meaningful team connection

An obvious one to land on. Non-work, social interaction really matters. It doesn’t need to eat into capacity – it can be a quick 30 minute social on a regular cadence. But that time makes all the difference for building connection long term and ultimately improving collaboration. Don’t drop it for ‘more important’ calls – make the time for it, and you’ll see the reward the next time you run a workshop where all the participants know and trust each other.

And if you can, where logistics allow, make that connection face to face, because meeting in person is a short cut to the kind of relationship that take remote communication a long time to build.

Subscribe now

The one that ends cleanly. Action items assigned, next steps clear, everyone nodding. These days, an AI-generated summary hits your inbox within about three minutes. It’s crisp, accurate, all the key points captured. Boom.

But somehow, despite all of that, nothing actually changes.

A concern surfaces in a Teams message that never came up in the room. An email lands the next morning that slightly repositions what was agreed. Someone messages you: “I’m not sure that’s actually decided yet.”

Sound familiar? I’d be surprised if it didn’t.

Here’s the thing though; we all know this happens. The question is why we keep acting like the meeting is the only moment that matters.

The meeting is not where decisions get made

I’m not trying to be flippant. Of course, some decisions get made in meetings. However, most meetings aren’t where decisions happen. They’re where decisions get announced.

Or performed.

Or deliberately left vague because naming them would force a confrontation nobody was ready for. Yup, I had at least one of those meetings yesterday too!

Think about the last big decision in your org. Where did it actually form?

In most cases, it was one of these:

• In a pre-meeting that you may or may not have been invited to. This is the conversation before the conversation. Someone did the rounds. Concerns were surfaced privately. Framing was tested. By the time everyone joined the call, the outcome was already in motion.

• In a conversation after the meeting, which was the real verdict on whether something actually landed. “That went well.” or “I’m not sure that resonated.” or “We should probably revisit that.” This is where alignment either consolidates or quietly falls apart.

• In a follow-up email that slightly repositions what was agreed. Usually not intentionally political, just someone trying to capture nuance. But whoever writes it shapes the record. Don’t underestimate this one.

• Or in a Slack thread where language gets tested before it goes anywhere official, and where the real temperature of a decision becomes visible if you’re paying attention.

So while the AI summary your team received after the meeting was useful, it can only capture what was said, not what was meant. It can’t tell you whose hesitation went unspoken, or whether the nodding heads in the room actually believed what they were agreeing to.

Why this makes some PMs invisible and others indispensable

There are two types of PM when it comes to meetings (and honestly, I’ve been both at different points in my career 😅).

Type 1 treats the meeting as the unit of work. They prepare for it, show up, perform well and move on. Often really articulate and confident in the room. And frequently confused about why their ideas keep not sticking. This was me earlier in my career.

Type 2 understands that the meeting is just one moment in a longer social process. They’re thinking about who needs to feel heard before the room fills. Or the one that needs quiet reassurance that the decision is aligned with their goals. They’re following up with the right person afterwards. They know that the conversation in the corridor after the session is often where the real decision gets made.

Here’s an important reframe that in my view helps you move from being a type 1 to a type 2. Type 2 isn’t ‘political’, or ‘playing the game’ but simply being cognizant of how decisions actually work.

That’s a meaningful distinction. Because a lot of the behaviour I’m describing gets labelled as “playing politics” which gives it a slightly grubby feeling. But it isn’t. It’s reading the system correctly, which is an important skill.

Three things worth changing from tomorrow

I don’t want to just leave you with the observation so here are a few practical shifts:

Prepare differently.

Before an important meeting, the question isn’t just “how do I make my case?” It’s “who in that room hasn’t been heard yet, and what would make this decision actually hold once we’re done?” Those are different questions, and the second one is usually more valuable.

Don’t mistake silence for agreement.

The people who say least in the meeting are often the ones whose buy-in matters most; and the most likely to share their actual view somewhere else, later, when it’s harder to address. If someone has been quiet, follow up.

Treat the follow-up as part of the decision.

What you say, to whom, and in what order after the meeting is where alignment either consolidates or quietly dissolves. This is not admin. This is the work.

The meeting was never really the point

The most important decisions forming in your organisation right now are probably not happening in any scheduled session.

They’re forming in the pre-meeting that happened this morning. In the quick note sent after last week’s planning session. In the follow-up someone is drafting right now.

The PMs who understand this don’t necessarily talk more or perform better in meetings. They just invest their attention differently: in the spaces around the formal moments, where the real work is still happening.

Next time you leave a significant meeting, try asking a different question. Not “did that go well?” but “where is this decision still unfinished?”

When a team gets access to AI-accelerated development, something predictable happens: output increases. Features appear on staging daily – and engineers previously blocked by capacity constraints start building at the speed of their curiosity. But that creates challenges around focus. Imagine requests from senior stakeholders: now there’s no reason for these not to go live almost immediately, irrespective of Product-Market Fit or customer sentiment. Similarly, design simplicity may get lost when everything is possible.

This is what velocity without progress looks like. The accelerator is slammed down hard, but the distance between where the product is and where it needs to be doesn’t really change. While AI compresses execution risk in the product development lifecycle, the organisations that benefit most will be those that connect their new speed to a clear sense of direction. Ultimately, outcomes still matter most.

Changing the risk profile

Marty Cagan’s four risks: value (will customers want this?), usability (can they figure it out?), feasibility (can we build it?), and business viability (does it work for our business?) provide a useful lens for what actually shifts in the new world (1):

Feasibility compresses

For many teams, feasibility was already the least dangerous of the four risks but consumed a disproportionate share of attention because it was the most visible constraint. But now, the cost and time to produce working software drops significantly, especially for new features and well-scoped work. Things requiring a sprint or two of engineering effort can be prototyped in hours, and the default answer to, ‘Can we build this?’ is increasingly, ‘Yes – and quickly.’

Usability shifts

Faster access to working software means usability issues surface in real usage rather than prototype testing. The risk does not disappear: it migrates from the lab to the wild. Faster, but messier.

Value and viability are reweighted

No amount of execution speed changes whether you are solving a real problem for real people, whether customers care enough to engage, or whether the solution fits your business model. These risks sit upstream of execution and they are the ones that determine Product-Market Fit.

But speed does change two things about them:

• First, you have more opportunities to test value assumptions through real usage, provided your experimentation and customer research can capture what you learn.

• Second, the consequences of not addressing these risks go up. Faster execution produces more evidence of strategic weakness, more quickly. Ship three features in a month that nobody uses and the pattern is hard to ignore.

Value and viability risk are not untouched by AI-accelerated development. They are reweighted: easier to validate if you are disciplined, more painfully exposed if you are not.

The shift in prototyping shows this clearly. Traditional prototyping existed partly because building real software was expensive, so it was simulated to test assumptions cheaply. When building becomes cheap and coding is driven through conversation with agents, this becomes far less necessary. Teams that once spent weeks crafting high-fidelity prototypes in Figma are increasingly vibe coding working software or designing directly inside tools like Cursor, so design outputs are now the product, not just a simulation. Yet the need to understand what you are testing and why remains: understand the problem, understand the customer, understand what success looks like.

From circular motion…

Slow delivery cycles have quietly acted as a safety net for weak strategy. When it takes a quarter to ship a feature, there is time to retrofit a narrative, adjust positioning, and claim alignment that was never really there. Nobody notices the rationale was thin because by the time the feature lands, the context has moved on enough to obscure the gap.

Rapid execution removes that buffer. If your strategy is vague, the gap between what you are shipping and what you should be shipping becomes visible almost immediately, and at volume. Leadership starts asking why the hit rate is low, and the answer is almost always the same: the team was building solutions to problems that were not well enough understood, or the definition of success was loose enough to accommodate any outcome.

A list of initiatives is not a strategy. Neither is a growth target. What teams need, more than ever, are guiding principles, clear positions on where to play and where not to, and high-level actions connecting business objectives to product decisions. When delivery was slow, you could get away with a vague destination and course-correct along the way. When delivery is fast, you build confidently in three different directions at once and get nowhere fast.

… To forward motion

If the danger is velocity without progress, the opportunity is a tighter feedback loop between strategy and execution. More frequent shipping generates more frequent signals about whether your strategic assumptions hold, but only if you are set up to capture and process them. Get this right, and each cycle of building, releasing, and learning brings you closer to Product-Market Fit rather than further from it.

Gate proportionally to risk

Not everything needs the same level of rigour. If the primary risks are feasibility or usability, adopt a build-first mindset and evaluate in production. If value or viability are genuinely at stake, do more discovery and customer research before committing. Be pragmatic and prevent product management becoming a bottleneck while maintaining discipline where the stakes are high.

There’s a middle path too, and one that becomes increasingly practical when working software is cheap to produce: beta groups and early-access cohorts let you put real software in front of real customers before a full release. Instead of testing assumptions against prototypes or mock-ups, test them against the actual product with people who can tell you whether it solves their problem.

The key is honesty about which category a piece of work falls into. Left ungoverned, teams will default to treating everything as low-risk, because building is now the path of least resistance.

Set the direction, don’t police the output

Talking about the ‘why’ with cross-functional colleagues has never been more important.

For work that does not need gating, PMs can keep things directionally sound by curating the strategic context that surrounds development. Product strategy documents, feature inventories, and experiment framing templates can live alongside the code, accessible to both developers and their AI coding agents. This does not prevent anyone from building, but it creates a shared reference point when someone asks, ‘Should we be doing this?’

When the pace of change accelerates, people outside the immediate product team need to understand the reasoning behind what is being built too. If the only thing that scales is output, you create confusion across the organisation.

Build the learning into the building

For features that go straight to build, the discipline of experimentation needs to live inside the development workflow, not upstream:

• What hypothesis does this feature test?

• How will we know if it is working?

• What are the criteria for keeping, iterating, or removing it?

If these questions are answered alongside the code, speed and rigour coexist. Without them, features and concepts accrue that are individually harmless but collectively incoherent. Building may be cheap, but maintaining, supporting, and explaining a growing product surface is not (2).

Your analytics need to keep pace with your release frequency too. If you are pushing to production weekly but reviewing metrics monthly, you have created a gap that swallows the advantage of speed.

Faster feedback

This is where the compounding effect lives. Most organisations are not short of data. What they lack is the ability to turn it into something actionable quickly enough to inform the next round of decisions. If AI can make building faster, it can make learning faster too, and it is the learning that determines whether speed translates to progress. Tools like Claude can compress the synthesis step, taking experiment results, usage data, and customer feedback and turning them into insight filtered through your product strategy. Not just, ‘Here is what happened’ but also, ‘Here is what this means for our assumptions.’

Yet none of this works if strategy is treated as a fixed document reviewed once a year. A two-way dialogue between execution and strategy is needed too:

Execution anchored through strategy. Strategy informed by accelerated learning.

What this means for product management

In an AI-accelerated world, the PM’s most important contribution is maintaining strategic coherence across a faster cycle: keeping the team pointed at problems that matter, ensuring customer insight feeds into direction, and making sure the product stays focused on user needs and business outcomes rather than on whatever is fun to build. This requires PMs to be deeply embedded in the team, not sitting above it writing specs that arrive too late to be useful.

This shift is already under way. In some organisations, engineers frustrated by the pace of product decision-making are going directly to customer recordings and building from what they hear. That is a rational response to a bottleneck – and PMs who define their value by what they prevent will find themselves increasingly worked around. But PMs who bring strategic grounding, customer centricity, and a relentless focus on whether the product is getting closer to solving real problems will find their contributions embraced.

The binding constraint in product development is no longer how fast you can build. It is how clearly you understand what you are building towards, and whether your organisation learns fast enough to steer. AI has removed one bottleneck. Are you ready for the ones it has revealed?

Sources

1. Cagan, M., Inspired: How to Create Tech Products Customers Love

2. McKinsey & Company, ‘How an AI-enabled software product development life cycle will fuel innovation’, February 2025: https://www.mckinsey.com/industries/technology-media-and-telecommunications/our-insights/how-an-ai-enabled-software-product-development-life-cycle-will-fuel-innovation

Finding Product Breaks valuable? Please consider sharing it with your friends - or subscribe if you haven’t already.

Subscribe now

Every organisation runs on content: strategy decks, product specs, pricing rules, brand guidelines, compliance policies, approval authorities. Most of this is created for people: to persuade, to inform, to cover a base. It lives in slide decks, PDFs and hidden SharePoint folders and, most importantly, in the heads of experienced staff who know which bits matter and what can be quietly ignored.

This has always been inefficient, but it’s sort of worked – because we humans are great at navigating ambiguity. We bridge the gap between documented and enacted reality every day, informally, expensively, but well enough.

Agentic AI changes the economics of this gap. When software can act – recommend a product, adjust a price, escalate a complaint – the content it operates on stops being background reference and becomes operational input. And operational input that is ambiguous, scattered, stale or contradictory produces operational outcomes with the same qualities, only faster and at scale.

This difficulty runs deeper than messy documentation. As I argued in the first post in this series, many organisations do not have clearly articulated, actionable strategies. If the strategic layer is vague, the policies and procedures beneath it cannot ladder up coherently – so the content that agents act on is unanchored.

Levels of content

An agent making a decision at the point of action needs content at several levels, each serving a different purpose:

• Strategic intent: The choices and trade-offs that define how an organisation competes. Are we optimising for growth or Customer Lifetime Value (CLV)? Where do we accept higher costs (and where don’t we)? This is the ‘why’ and the ‘what we have decided.’

• Policy and rules: The codified constraints that govern how intent translates into action. Legal and compliance requirements, discounting authorities, escalation thresholds. This is the ‘what is permitted?’

• Domain knowledge: The reference material an agent needs to reason well. Product catalogues, customer segmentation models, competitor positioning, regulatory context. This is the ‘what do we know?’

• Operational parameters: The specific values, thresholds and configurations that tune behaviour. A maximum discount percentage, a risk score threshold for escalation, a response time target. This is the ‘how tightly?’

The challenge is not that this content doesn’t exist: it almost always does, somewhere. But it was never designed to be retrieved, applied and maintained as a system. The layers of content – and thinking – within an organisation are rarely connected in ways that agents can navigate.

Most organisations are, in practice, inverted: thin and vague at the top, thick and specific at the bottom. Lower-level content, like system configurations, compliance checklists and process documentation, often exists in reasonable detail because it has to. Someone needs to pass an audit, configure a platform, or onboard new staff. These decisions are small, low ambiguity and often easy to codify. The gaps tend to widen as you move up the stack. Strategic intent and the policies that should ladder from it are where ambiguity has longest been tolerated – but where agents need clarity most.

Why does this matter? Because agents are sensitive to misalignment between layers in ways humans are not. A human salesperson can hold ‘our strategy is CLV’ and ‘my bonus is on conversion’ in their head simultaneously and navigate the contradiction through judgement (though I’ve yet to meet one who won’t lean towards the bonus). An agent given contradictory context from different layers will either pick one arbitrarily or behave inconsistently. Conflicts between layers need to be resolved before the agent encounters them, not after.

Making this concrete: a D2C sales agent

Meet Tarra, a direct-to-consumer skincare brand. She has premium positioning, a subscription-led model, and roughly fifty SKUs across three tiers: an entry range (£15–25), a core range (£30–50) and a clinical range (£60–90). The team at Tarra are building an AI sales agent: a guided shopping tool that recommends products based on skin type and concern, handles objections about price and ingredients, and can apply promotional discounts autonomously within defined bounds.

The strategic tension across horizons

This quarter, Tarra’s head of growth (Zara) wants to hit a revenue target. Zara is pushing the agent to lead with the clinical range, where average order value is highest, and to offer 15% first-purchase discounts to reduce basket abandonment. A sales agent optimised for Zara’s context will push premium, discount early, and close fast. It will look productive.

Over the year, Tarra’s retention lead (Clara) cares about CLV. Her data shows that customers who start with the core range and build a routine are three times more likely to subscribe than those who enter through a discounted clinical purchase. The same agent, given Clara’s context, would recommend the core range first and hold back on discounting – a smaller basket today, but a customer who returns.

Over the longer term, the brand matters. Tarra’s positioning depends on trust: dermatologist-backed formulations, honest ingredient communication, no hard sell. An agent that aggressively upsells the clinical range to someone with a simple hydration concern, or makes efficacy claims the formulations cannot support, erodes that trust in ways that do not show up in this quarter’s dashboard.

These are the same tensions that human sales teams navigate daily, mediated by management, culture and experience. But the agent is not in the room when Clara, Zara and the rest of the leadership subtly shift towards a retention focus on their weekly call. Nor is it standing by the watercooler. It will do what its digital context tells it to do.

Content as a managed operational asset

This is the shift that matters, and it is easy to understate. It is not a question of documentation, but a change in how organisations manage, maintain and value their content – because in an agentic context, that content is not reference material for people. It is the context that fuels the business at scale (1).

A useful analogy is data management. Two decades ago, most organisations had data scattered across operational systems – CRMs, ERPs, spreadsheets, email. The data existed, but it was not usable for analytics or decision-making at scale. The response was ‘ETL’: Extract data from source systems, Transform it into a consistent and queryable structure, and Load it into a warehouse where it could be used.

Organisational content now faces a similar transition. The raw material – strategy documents, policies, product data, operational rules – needs to be extracted from its native formats, transformed into representations that agents can retrieve and apply, and maintained in systems where it can be versioned, governed and kept current. We need to create machine-usable content warehouses.

Source content vs. derivative content

The ETL analogy helps surface what may be the most underestimated risk in building a content layer: organisations are full of derivative content, and much of it looks authoritative.

Let’s return to Tarra. The board deck contains the strategic positioning. But the sales enablement team has produced a pitch deck emphasising the clinical range, because that is what retail buyers want to hear. The customer service team has a one-pager that summarises the returns policy in friendly language, omitting several conditions. The marketing team has rewritten the ingredient claims for social media, softening the regulatory caveats. Each is a retelling – filtered, simplified, sometimes unconsciously editorialised.

They have created derivative content. If the agent’s context layer is built from these retellings rather than the sourcematerial, it inherits multiple, partial interpretations of reality.

Nobody at Tarra deliberately created contradictory information. Each retelling shifts the emphasis, not through malice or neglect, but through the entirely normal process of human communication. This gradual drift between what was decided and what gets retrieved is one way content layer misalignment enters the system. Derivative content is necessary for human audiences: it is not a safe foundation for agents that need to act. The content layer must be built from the ground truth: actual strategy choices, actual policies and brand guidelines, actual product data.

Library, instruction manual, charter

Not all content functions the same way – and treating it uniformly is a mistake. Some content functions like a library: reference material the agent looks up as needed. Product specifications, regulatory requirements, competitor positioning. The agent queries it, uses what is relevant, and moves on. In technical terms, this is where retrieval-augmented generation (RAG) – systems that search a knowledge base and feed relevant content to the agent at the point of decision – does its work.

Other content functions like an instruction manual: procedural knowledge that tells the agent how to act in specific situations. Tarra’s discounting rules, escalation logic, the decision of when to recommend the clinical range versus the core range. This content does not just inform – it directs (in my next post, I will cover how this type of content gets encoded as callable Skills).

Lastly, some content functions like a charter: the strategic choices and trade-offs that express what the organisation has decided and why. This is the content that resolves conflicts between competing objectives – the reason Tarra’s agent should recommend the right product for the customer’s skin concern rather than the highest-margin SKU, even though this quarter’s revenue target says otherwise.

Each content type has different characteristics. Libraries need breadth, freshness and good retrieval. Instruction manuals need precision and testability. Charters need authority and clarity. Treating them all the same – and dumping everything into a knowledge base and hoping retrieval sorts it out – is the content equivalent of loading raw operational data into a reporting tool and wondering why the dashboards are wrong.

Format and structure: practical considerations

If content becomes operational input, the way it is structured matters more than it ever has. A few principles are emerging, though this is an area where practice is still ahead of consensus.

Density and fidelity: There is a tension between completeness and usability. A 47-page compliance policy might contain everything an agent needs, but retrieving from it will return noise alongside the signal. Yet a one-paragraph summary will be clean but miss the edge case that matters when a customer invokes a specific regulatory right.

The practical answer is layering. Strategic choices and trade-offs should be concise and opinionated – a page, not a deck. Policies need enough detail to be applied in specific situations but should be chunked by decision domain, not structured as monolithic documents. Product data should be structured with attributes that support the decisions agents will make, not written as marketing prose.

Structure over prose where the agent must act: For content that directs behaviour – decision rules, escalation logic, pricing constraints – structured formats (decision tables, rule sets, parameterised thresholds) produce more consistent results than natural language. An agent interpreting ‘use your judgement on discounting for high-value customers’ will behave variably. An agent applying ‘maximum 15% discount for tier-1 customers on the core range, 10% on clinical, approval required above these thresholds’ will not. These patterns translate well into machine-usable Python.

Prose is still appropriate for content the agent needs to reason about rather than execute: strategic rationale, brand positioning, customer context. But even here, concise and well-structured prose retrieves better and produces more consistent reasoning than long, discursive documents.

Chunking by decision, not by document: Most organisational content is structured by authorship. The compliance team wrote a compliance policy, the brand team wrote brand guidelines, the commercial team wrote a pricing framework. Tarra’s agent making a discounting decision needs fragments from all three. Content that is chunked by decision domain – ‘everything relevant to the discount decision’ – is more retrievable than content chunked by organisational origin. Interestingly, this suggests that Conway’s Law is in effect around agentic behaviour.

Carry metadata for governance: Every piece of content an agent uses should carry enough metadata to support auditability and make clear which version is current: who owns it, when it was last reviewed, and its authority (regulatory, commercial, advisory). This should not be considered an overhead: it is the mechanism that allows you to trace a bad decision back to the context that produced it, to distinguish a regulatory constraint from a preference someone introduced three years ago, and to support retrieval filtering and weighting – ensuring the agent searches the right subset of content and prioritises appropriately authoritative sources.

Keeping the content layer alive

Artefacts decay. For many organisations, building an optimised content layer will be hard, but maintaining it may be even harder.

Many of the rules used for knowledge base and information system management are still relevant, but with greater rigour around provenance, versioning, and evaluation:

• Ownership must sit with SMEs: Not in the AI team, not in a documentation function, but in the part of the organisation that makes the decisions the content encodes. If information ownership is delegated to people who do not live inside the decisions, the content layer becomes fiction.

• Version management needs rigour: When an agent makes a bad decision, you need to know which version of which content informed it. This is not perfectionism, but the minimum requirement for meaningful oversight. For the same reasons, provenance matters: without it, everything may look equally binding to an agent.

• Evaluation: The moment the content layer drifts from enacted reality, agent behaviour drifts with it – and unlike a human team, agents will not flag the contradiction. Detecting that drift requires systematic evaluation of whether the content agents are retrieving is still producing the outcomes you expect – a discipline that connects directly to the evaluation frameworks I have written about previously.

A practical starting point

Pick one bounded decision domain – something repeatable where the stakes are real. For Tarra, that might be the product recommendation decision: given a customer’s skin type and concern, which range does the agent lead with, what claims can it make, and what discounting is it permitted to offer?

Map what actually happens today (as opposed to what the policy says should happen). Write down the strategic choices that should govern the decision, the rules that constrain it, and the parameters that calibrate it. Identify the source content: not the sales deck or the internal summary, but the actual policy, the actual product data, the actual regulatory constraints. Structure that material so it can be retrieved and applied. Give it an owner who lives inside the decision.

You will learn more from making one decision domain machine-usable than from simply cataloguing twenty. And if you have ever built a data pipeline – extracted messy source data, transformed it into something queryable, and fought to keep it governed and current – the discipline will feel strangely, sadly familiar.

In my next post, I’ll look at Skills: how context becomes callable, testable procedures that agents can execute – and how teams need to organise to build and maintain them.

Sources

1. Anthropic – Effective context engineering for AI agents https://www.anthropic.com/engineering/effective-context-engineering-for-ai-agents

Finding Product Breaks valuable? Please consider sharing it with your friends - or subscribe if you haven’t already.

Subscribe now

Feature ideation & design: from shipping features to shaping behaviour

In AI products, the old habit of ideating discrete features breaks down quickly. What matters more is the behaviour you’re trying to unlock and the outcome you want users to achieve, because the “feature” is often a fluid interaction between model, UI, and user intent. While being outcome-oriented has always been at the heart of product thinking, how users get to their desired outcome using AI products is less clearly defined. PMs and design teams have to shift from spec’ing static functionality to designing systems that guide, constrain, and learn from user behaviour over time. That means spending less time perfecting high-fidelity designs and flows up front and more time prototyping end-to-end experiences, testing how people actually prompt, correct, and rely on the system, and baking in guardrails where the model is likely to go off-script. Creating and applying solid experience guidelines here is crucial, ensuring safety, transparency and trust in our interactions.

Workflow mapping: from linear funnels to human–AI loops

As PMs, we’re generally used to clean funnels and deterministic flows: user does A, system does B, outcome C happens. However, in AI products, real workflows become loops, with humans prompting, the model responding, humans correcting or steering, and the system adapting. This becomes even more critical with agentic AI, where the system is no longer just responding but planning, taking actions, and chaining steps across tools and contexts. Now the “workflow” includes planning, tool use, intermediate decisions, and actions taken by the system on the user’s behalf, often across multiple steps and agents. The shift for PMs and dev teams is to move from mapping happy-path funnels to designing and stress-testing autonomy boundaries: what the agent is allowed to decide, where it should retrieve information from, when it should ask for confirmation, how it exposes its reasoning or plan, and how users can interrupt, correct, or roll back actions when things go wrong. For more on the topic of agentic AI workflow mapping, take a look at my colleague Sophie’s recent post.

Evals: from QA checklists to continuous measurement of usefulness

In classic software, QA and acceptance criteria are about whether the feature works as specified. In AI products, determining whether something “works” is less clear, and model behaviour drifts with data, prompts, and use cases. The shift here is from one-off validation to ongoing evaluation of usefulness, reliability, and failure modes in the real world. PMs need to think about evals as a product surface: what signals actually reflect user value, what bad outcomes matter most, and how quickly can the team detect and respond to regressions? This pushes teams to invest early in lightweight eval pipelines, user feedback loops, and qualitative review, rather than treating evaluation as a late-stage, purely technical concern. As above, workflow mapping is a great way to define where evaluations should be applied throughout the journey, to break them down into more discrete, testable chunks. My colleague Tim recently shared this great article on lean thinking for evals here.

Organisational readiness: from shipping teams to learning organisations

Finally, building AI products isn’t just a product or engineering shift; it’s an organisational one. The old model assumes stable requirements, predictable delivery, and clear ownership boundaries. AI work is messier: uncertainty is higher, iteration cycles are shorter, and the “right” answer evolves as models and user behaviour change. The shift PMs and leaders need to make is from optimising for execution to optimising for learning. That means creating space for experimentation, normalising that some bets won’t pan out, and aligning incentives around insight gained rather than features shipped. Teams that can learn fast, adapt their mental models, and collaborate across product, design, data, and policy will outpace teams that treat AI like just another tech stack upgrade.

My overarching reflection is that product thinking in this space is less about finding the perfect spec and more about designing for uncertainty: making intent explicit, constraining autonomy thoughtfully, instrumenting what “good” looks like, and building teams that can respond when reality doesn’t match the plan.

Most organisations have spent decades building systems that tolerate ambiguity. Unclear strategy? People negotiate. Contradictory priorities? Managers mediate. Undocumented trade-offs? Experienced staff just know. Sometimes, this works – expensively, slowly, and with a lot of meetings. But it only works because humans are remarkably good at filling gaps that the organisation has never made explicit.

Agentic AI offers something genuinely new: systems that can plan, retrieve context and act within workflows at a speed, consistency and scale that human coordination cannot match. But that promise has a prerequisite. Every gap in your organisational clarity becomes a gap in its performance (1).

In this series of posts, I explore how we might close these gaps.

From narrative to runtime input

These are not the AI assistants most organisations have started with: the tools that summarise documents, draft copy or speed up analysis. Those are useful, but they operate under human micromanagement. Someone still decides what to do. Agentic systems offer a different type of shift. Instead of providing, ‘Write me an email’ level instructions, we expect agents to, ‘Triage this queue, apply policy, take the next step and escalate only when you should.’

Adoption will not be uniform: some organisations will move slowly for valid reasons – risk, complexity, strategic fit. Others might just be slow to respond. But for those that do embed agents into day-to-day work, the practical constraint becomes hard to avoid:

Once software can act, the quality of the organisational context it operates with becomes a primary driver of value, safety and consistency.

Think of it this way. Strategy and organisation design have traditionally been background narrative: the deck, the offsite workshop, the inspirational memo. Agentic AI turns that narrative into a runtime input. If that input is unclear, contradictory, or out of step with how decisions actually get made, agents will either be constrained to superficial assistance or behave in ways that are locally plausible and globally misaligned.

Strategy as a usable system of choices

For agents to be strategically aligned, we need to understand what strategy is.

A useful starting point is Richard Rumelt’s critique of ‘bad strategy’: organisations often present a mixture of goals, slogans, and initiatives as strategy, without a central logic that links the challenge to a coherent approach and set of coordinated actions. Rumelt’s ‘kernel’ framing – diagnosis, guiding policy, coherent action – is valuable precisely because it makes strategy operational. It asks what problem you are solving, what overall approach you will take, and what mutually reinforcing actions will follow (2).

When strategy is weak, a familiar pattern emerges: trade-offs remain implicit, operational decisions get pushed into meetings and escalation, tactical work fragments into locally rational activity. The organisation stays busy, but coherence becomes expensive.

This matters for agentic systems because agents need to interpret situations and choose actions repeatedly. If strategy is not expressed as a set of choices and trade-offs, then every decision becomes interpretive. Humans can often manage that through judgement and informal alignment. Agents cannot infer intent from status, history, or politics unless those dynamics are made explicit in the context you provide.

Documented vs enacted decision-making

But writing something down doesn’t make it true.

Just as many organisational strategies are simply to-do or wish lists, most organisations already have artefacts that claim to describe how decisions are made: governance diagrams, approval routes, escalation paths, decision-rights matrices. The difficulty is that these artefacts frequently describe the intended organisation rather than the enacted organisation. Under pressure, decisions often flow through informal influence, relationship networks, and situational overrides.

This is not a moral failing – it is an adaptive response to ambiguity. When the formal system is unclear or slow, people use workarounds: trusted individuals mediate conflict, senior stakeholders intervene, and compromises keep things moving. Over time, those workarounds become the real system.

Decision-rights models illustrate the issue well. A RACI chart can look tidy while the day-to-day reality is shaped by informal vetoes, escalation habits, and ‘who has to be kept comfortable’ rather than who is nominally accountable. McKinsey & Company make a similar point in its critique of RACI: it can confuse who decides and encourage bureaucracy rather than improve decision quality (3).

Humans can navigate this gap because they constantly update their internal model of how things really work. Agents won’t succeed here unless given a maintained and machine-usable representation of enacted decision logic.

Making this concrete: contact-centre triage

A contact centre is a useful place to ground this. On paper, triage is an operational question: who goes first and where does the work go. In practice, it is a daily enactment of strategy.

Take two organisations with different strategic positions:

• A premium, trust-led organisation that wants customers to feel looked after and is willing to spend more to protect trust.

• A cost-led organisation which competes on price and must keep cost-to-serve under tight control.

Now imagine the same inbound contact: ‘I want to cancel.’

A human team will often treat that differently depending on where the organisation is trying to win (4). The premium organisation might invest time to retain and recover relationships more than the cost-led organisation, which accepts more churn and minimises handling time. Both responses can be strategically correct, but only if they are deliberate.

An agent cannot infer that orientation from a slogan. If the only goal it is given is ‘reduce backlog’ or ‘improve handle time’, it will optimise for speed and closure. If it is given vague instructions like ‘be customer first’, it will likely hesitate, escalate too often, or behave inconsistently. Experienced contact-centre staff do this intuitively: they absorb the culture, read the signals, know when to bend a rule – even find ways to massage tensions between high-level objectives and on-the-ground policies. This is harder for agents to do.

Why context quality becomes a first-order capability

When people talk about agents, they tend to focus on model capability: which model is smartest, fastest, cheapest. In practice, the harder work is almost always in the surrounding system: what the agent can see, what it’s permitted to do, the outcomes it’s optimised for – and who is held accountable for the results.

This is where ‘context engineering’ becomes a useful framing. It is the discipline of structuring and maintaining the information an AI system uses at the point of decision. The difference is not just ‘more documentation’. It is the difference between giving a system a filing cabinet and giving it a brief it can act on.

Our contact centre agent needs more than the category of the ticket. It needs to know how the organisation makes trade-offs. That typically includes: what outcomes matter, what constraints are binding (refund limits, regulatory rules, brand posture), who can decide what (including override conditions), what actions are permitted without approval, and what gets logged and reviewed. Without those inputs, the agent defaults to what it can reliably optimise: measurable proxies like closure rate or handle time.

If you want an agent to operationalise strategy, the strategy needs to be usable at the point of decision. This rewards organisations who express their strategy in meaningful terms, generate policies that track with this strategy, and reward behaviours that ‘live’ this strategy.

The advantage thesis

None of this implies that organisations without strong strategy cannot benefit from agents. Narrow deployments can deliver value in many environments.

But there is a ceiling. As autonomy increases, organisations with clearer strategy and legible decision loops can safely widen the envelope of what agents do. They can grant bounded autonomy with confidence because intent, constraints and escalation logic are explicit enough to be executed and audited.

Organisations that rely heavily on tacit coordination – the quiet craft, the read-the-room culture – can still adopt agents, but they will end up with conservative usage: agents that suggest, summarise and route, because anything beyond that requires a level of clarity the organisation has not yet built.

A closing test

Pick one bounded decision domain – something repeatable and consequential – and ask whether you can describe, in a way that matches enacted reality:

• The intended outcome and the trade-offs that constrain it.

• What information counts as evidence.

• How this activity reflects higher-order goals.

• Who decides, who can override, and what escalation looks like in practice.

• What actions are permitted, and what must be logged.

• How the loop is reviewed and updated as the organisation changes.

If those questions are hard to answer, the constraint is unlikely to be the model. It is your organisation’s ability to express intent and decision-making in a form that can be executed consistently, first by people – and then by agents.

So, what does ‘machine-usable strategy’ actually look like? That is the subject of my next post.

Sources

1. Anthropic – Effective context engineering for AI agents: https://www.anthropic.com/engineering/effective-context-engineering-for-ai-agents

2. Lenny’s Podcast – Richard Rumelt:

Lenny's Newsletter

Good Strategy, Bad Strategy | Richard Rumelt

Brought to you by…

Listen now

2 years ago · 246 likes · 1 comment · Lenny Rachitsky

1. McKinsey – The limits of RACI, and a better way to make decisions: https://www.mckinsey.com/capabilities/people-and-organizational-performance/our-insights/the-organization-blog/the-limits-of-raci-and-a-better-way-to-make-decisions

2. Roger Martin – Strategy choice cascade: https://rogerlmartin.com/thought-pillars/strategy https://hbr.org/books/playing-to-win

Finding Product Breaks valuable? Please consider sharing it with your friends - or subscribe if you haven’t already.

Subscribe now

LLM-driven products are sensitive in ways that are easy to underestimate because the probabilistic nature of these systems fundamentally changes the notion of QA. A small prompt change, model upgrade, or retrieval adjustment can shift behaviour in ways that aren’t obvious in code review and can’t be caught reliably through spot checks. OpenAI’s guidance is direct on this: traditional testing methods are not sufficient on their own for generative AI, ‘quality’ is harder to pin down (1).

Here, the mindsets and specific techniques associated with Lean provide useful mental models for Product Managers – and product teams as a whole. Not because AI development is manufacturing in the traditional sense, but because Lean focuses on standards, flow, the visibility of problems, and continuous improvement. This connects closely to the emergence of Evals as best-practice.

What Evals are (and what they’re not)

An Eval is a repeatable way to measure whether an AI system meets defined quality criteria across representative scenarios. Most practical Eval setups include:

• Scenario sets: realistic inputs with enough context to represent real usage.

• A rubric / scoring method: human review, automated scoring, or a hybrid. Many teams use model-graded Evals (LLM-as-judge) for scale, with human spot checks for calibration.

• Baselines and thresholds: what “good” currently looks like, and what counts as a regression.

• A harness: the code and infrastructure to run these checks repeatedly.

The point is not to build a perfect measurement machine. But we do want a quality system that sits inside day-to-day delivery, with shared measurement that enables sensible trade-offs and continual learning. Anthropic’s overview is worth reading because it makes clear how difficult (and important) evaluation is in practice (2).

A simple way to keep your thinking grounded is to name the quality dimensions you actually care about. For many products, this boils down to:

• Task success (did the user/agent achieve the goal?)

• Factuality/grounding (is it accurate and properly supported?)

• Safety/compliance (does it stay within policy and regulation?)

• Tone/brand (does it communicate appropriately?)

• Latency (is it fast enough?)

• Cost (is it economically viable at scale?)

Note: Many of these criteria align with the Usability, Feasibility and Viability lenses we commonly apply to product development. Product-Market Fit and other desirability-type considerations are also crucial, but their connection to Evals might be slightly looser.

Let’s make this tangible

Imagine a contact-centre AI agent that reads an incoming customer email and decides what to do next.

• Baseline: it classifies the request (refund, delivery issue, account change), retrieves the relevant policy, drafts a reply, and chooses one of two paths:

  • Automate (send the message and, where appropriate, trigger a simple backend action) or,

  • Escalate (route to a human supervisor with a summary and recommended next step).

• Change: a few weeks later, you tweak the prompt to make the agent more “helpful” and reduce escalations.

• Regression: the agent starts taking confident actions in borderline cases—promising refunds or processing changes that should have been escalated—because the decision threshold for escalation has effectively shifted.

The outputs still look good in isolation, but the failure shows up in the workflow: the wrong cases get automated, and humans see them only after customers complain. This is the pattern Evals are designed to prevent: a shared standard for ‘automate vs escalate’, fast signals when that boundary drifts, and a habit of turning these incidents into permanent tests.

Lean > QA

So, Evals often look like QA: test cases, regression suites, CI gates. And this similarity is helpful.

However, the difference is operational. Where many software tests are binary and stable over time, Evals often are not. They measure degrees of quality across multiple dimensions, in a system whose behaviour can shift with prompts, data, retrieval, or model changes. Google’s ‘ML Test Score’ rubric captures the broader idea: production readiness for ML systems depends on ongoing evaluation and monitoring, not just pre-release testing (5). This is where Lean thinking comes in.

So, the framing is:

• QA mechanics help you implement Evals.

• Lean helps you run Evals as a quality system.

Mapping Lean principles to Evals

1) Standard work: define “good”, link it to outcomes, and keep it visible

In Lean, standard work is the baseline that makes improvement possible. Without a shared definition of ‘normal’, you can’t reliably spot abnormality.

In Eval terms, ‘standard work’ is:

• Rubrics: concrete definitions of ‘correct’, ‘helpful’, ‘safe’, and ‘on-brand’.If you use a model to grade outputs, treat the grader prompt as part of the standard. If the grader drifts, your standard drifts.

• Scenario classes: the request patterns you care about, including edge cases and known failure modes.

• Thresholds: what must not regress and what is ‘good enough to ship’.

To make this usable inside a product team, communicate it like any other product standard:

• Put the rubric and scenario set somewhere obvious (alongside design principles, accessibility standards, API contracts).

• Make it part of normal rituals: sprint planning (‘What Eval coverage changes?’), release reviews (‘What moved vs baseline?’), and incident reviews (‘What scenarios should we add?’).

• Tie it to outcomes you are working towards and make things clear. In the contact-centre agent example, that will include metrics like: First Contact Resolution (FCR), escalation rate and after-call work (ACW) driven by incorrect automation. CSAT and key complaint themes, and cost per contact (including model and tooling costs) should also be measured.

2) Andon: make problems visible early, and keep the feedback loop fast

Andon is about surfacing issues quickly so they can be addressed before they spread (4).

In Eval terms, the idea is simple: regressions should be visible soon after they are introduced. But there is a practical constraint: an Andon cord is useless if it takes an hour to pull. So, treat speed and cost as product requirements for your Eval suite:

• Developer-loop signals (fast and cheap): run in minutes on every meaningful change. These might include schema/JSON validation, tool-call correctness checks, simple heuristics, small-model graders, or targeted ‘golden path’ scenarios. These should be fast and low cost per run.

• Production signals (ongoing): sampling queues, drift indicators, escalation spikes, and ‘thumbs down’ themes, reviewed on a regular cadence.

The key PM responsibility is the operating agreement: who responds when the signal goes red, what happens next, and how quickly you turn that signal into a prioritised countermeasure.

3) Jidoka: decide what stops the line, without creating Muda (waste)

Jidoka is commonly summarised as ‘build in quality and stop when abnormality appears,’ so defects do not keep flowing downstream. The AI version is a tiered approach that protects outcomes without creating muda (waste), including wasted developer time and unnecessary API spend (3). Here, we can separate cadence (when you run Evals) from severity (what they mean).

A model cadence might be:

• PR gate (fast): runs on every meaningful change.

• Nightly regression suite (comprehensive): wider coverage, slower, more expensive.

• Pre-release suite (targeted + comprehensive): used when risk is higher or the change is large.

While severity might differentiate between:

• Hard blockers (stop-the-line): unsafe behaviour or policy violations, clear factuality regressions in sensitive domains, consistent failure on core journeys, tool misuse with real-world consequences.

• Warnings (triage): minor tone drift, small verbosity changes, modest latency/cost drift still within budget.

For example, in our contact centre, inaccurate agentic actions that trigger issues downstream might be hard blockers; while true tonal issues in communications might considered less urgent. Lean is useful because it makes the trade-off explicit: you are balancing flow, quality, and cost deliberately.

4) Kaizen and poka-yoke: make learning cumulative and reduce repeat failures

Kaizen is continuous improvement; poka-yoke is mistake-proofing. In Eval terms, the loop is straightforward:

1. If a failure matters, capture it as a scenario.

2. Add it to the Eval set so it cannot recur silently.

3. Implement a countermeasure (e.g. prompt/tool/retrieval/UX/guardrail).

4. Re-run Evals, update baseline and thresholds.

Synthetic data can make this stronger, but it needs guardrails:

• Start small: generate a limited number of variants.

• Promote only variants that resemble real user behaviour or plausible risk.

• Where possible, seed from anonymised production patterns so you do not drift away from reality.

This is how Evals become a maintained asset, rather than a static benchmark.

Do things change for agentic workflows?

No. While many AI products are now workflows: classify-retrieve-draft-check-act, many are increasingly ‘agentic’ – like the contact centre example, where the system can choose which steps and tools to use based on context (6)(7).

Agentic systems do not need a new Eval philosophy, but they do need clearer structure, because small component regressions can compound: a retrieval miss can turn into a wrong action, not just a slightly worse answer.

• End-to-end Evals (integration): did the assistant avoid promising refunds when the user is ineligible, while still resolving the query appropriately?

• Component Evals (diagnostic): did retrieval return the policy section that contains the eligibility rule?

A simple rule is usually enough: start with end-to-end coverage on critical journeys, then add component Evals only where traces show repeated failure. Over-testing every step is a common source of waste. LangSmith’s guidance on evaluation approaches is a good reference for thinking about step-level vs system-level evaluation (8).

Limits of the Lean mental model

The Lean analogy is useful, but please note that this is not a 1:1 mapping. For AI products:

• Quality is less objective: ‘tone’ and ‘helpfulness’ are harder to measure than a binary defect. This means Evals need calibration and periodic human review.

• Inputs are unbounded: user behaviour shifts, and Eval sets can go stale unless refreshed from real usage.

• Metrics can be gamed: you can optimise for a high Eval score while degrading user experience (Goodhart effects). This is another reason to balance End-to-end and Component level evaluation.

Applying this as a PM

The switch from traditional QA (and UAT) to Evals requires a new way of thinking. So, if Evals are not gaining traction in your team, use Lean as a sequencing tool:

1. Map the value stream: inputs → retrieval/tools → reasoning/steps → output → user/ business outcome.

2. Create standard work: rubrics, scenario classes, thresholds, and a calibrated grading approach.

3. Add Andon: fast developer-loop signals plus a production sampling/review loop.

4. Tier your gates (Jidoka): separate cadence from severity and protect core outcomes without excessive waste.

5. Run Kaizen: every meaningful failure becomes a test and use carefully chosen variants to prevent near-miss repeats.

The aim is not to add process. It is to make quality work cumulative, so you stop solving the same problems twice. Lean thinking has been transformative in manufacturing – and is central to good development practice. It will work well in the AI product world too.

Sources

1. OpenAI, “Evaluation best practices”: https://platform.openai.com/docs/guides/evaluation-best-practices

2. Anthropic, “Evaluating AI systems”: https://www.anthropic.com/research/evaluating-ai-systems

3. Toyota Motor Corporation, “Toyota Production System”: https://global.toyota/en/company/vision-and-philosophy/production-system/index.html

4. Toyota UK magazine, “Andon”: https://mag.toyota.co.uk/andon-toyota-production-system/

5. Google, “What’s your ML Test Score?”: https://research.google/pubs/pub45742/

6. Anthropic, “Building effective agents”: https://www.anthropic.com/research/building-effective-agents

7. IBM, “Agentic workflows”: https://www.ibm.com/think/topics/agentic-workflows

8. LangSmith, “Evaluation approaches”: https://docs.langchain.com/langsmith/evaluation-approaches

Finding Product Breaks valuable? Please consider sharing it with your friends - or subscribe if you haven’t already.

Subscribe now

Wondering what it actually is, and how it compares to generative AI? Me too. I spent my first week on an AI project Googling orchestration and deterministic workflows, trying to wrap my head around all the new terminology. Throw in ever-changing technology and rapid turnaround times, there’s a lot to contend with for a product manager.

In short, while generative AI is AI that says something, agentic AI is AI that does something. Typically, it performs a task based on an instruction (e.g. ‘Write my Product Breaks article’). This process might be broken down into a series of tasks, bringing in multiple agents. Agent 1: ‘Research agentic AI for my Product Breaks article.’ Agent 2: ‘Write a first draft.’ Agent 3: ‘Make updates to the draft based on my feedback’.

Here’s what I wish I knew before I started with agentic AI.

1. It’s all about the workflow

Agentic AI can be a complex puzzle to communicate, especially when it’s still a work in progress. Mapping out an agentic workflow is a way to provide a central and up to date view of how the product is being built. This is key to the team and wider stakeholders’ understanding, particularly around where and how agents and humans will interact.

To get started, Andrew Ng’s AI course has a really helpful section on creating a workflow, which is very similar to a user journey but with AI-specific inputs. Ask yourself:

• What are the steps a human would take in the process, and how you would make updates if things weren’t right first time? How many steps would you need?

• What are the key pieces of information you need the LLM to extract – and where should they get them from?

• What agents are involved?

• What platforms are involved, and at which stage in the process?

2. Keep it simple

Once the process has been broken down and there’s a shared understanding of where agents can be used, roles need to be assigned to them.

To make agents the most effective, they need a singular, simple job to focus on. This might mean that one step in the process requires several of its own agents, but that way they can become better at their jobs and are easier to refine.

Tip: Giving agents unique names (e.g. Email Writing Agent) helps to keep track of and reference them.

3. You’ll need to manage expectations

A huge benefit of AI is that development is quicker, and prompts can be updated as and when to rapidly improve the output. The downside of this is it can result in high expectations around what’s possible. This is understandable given the narrative about AI, but not necessarily realistic.

Bringing in agentic AI likely means automating a process with multiple people, teams, platforms and information sources. As PM, a useful way to communicate this may be that it offers a chance to test out what can be automated, which keeps expectations in check.

4. AI is only as good as the data it can work with

This is another reason that expectations must be managed; AI needs high-quality data to produce high-quality output. In fact, this is one of the main barriers to AI adoption in general. There are two main elements to this:

Accessibility

A clear set of rules for storage and naming convention, so the LLM knows exactly where to look for information.

Mapping

A good understanding of where the information needs to go, how it will get there, and what requirements it should adhere to if managing an integration. This would also influence what data is collected in the first place, and in what format.

While it’s still possible to prove out the potential of AI without this, proving out the value is likely to be much more achievable with high quality data in place.

5. What is true today could change next week

AI capabilities and the technology powering them are rapidly changing – which is both a blessing and a curse. There are constantly new features available, but often these are still in beta or soon to be updated again. This might mean that workarounds are required to get a product built quicker, even if not fit for the long term.

In summary, working with agentic AI was a big learning curve as a PM, but a lot of the experience itself wasn’t new. Being comfortable with constant change and the ability to explain complexity are fundamentals from product management which will serve us well.

But when I moved into Product Management, I suddenly had a big role in deciding what to build next. Learning about “product discovery” was dizzying. There was so much to consider.

All the discovery activities alongside building, shipping, and landing features were a lot to juggle. I struggled to chart a clear path forward or to communicate the wider plan.

But over the years I’ve learned some simple techniques that have helped me to regain the control that I enjoyed in my project management years. In this article I’ve set out the five frameworks that have helped me to tame the product discovery beast.

Exploring options

1. Opportunity-Solution Trees

Teresa Torres’s Opportunity-Solution Trees are a great way to shift mindsets from delivery to discovery. They provide structured collaboration and communication around product discovery and really help me to clarify my thinking.

Opportunity-Solution Trees encourage you to consider a range of opportunities that might contribute to the outcome that you’re aiming towards. Then they prompt you to consider the possible solutions that might deliver on those opportunities. (For opportunities, read a user need, desire, or pain point.) Breaking the tree down more helps you to track assumption tests. These tests help you to assess how much the proposed solution can help you reach your goal.

Using opportunity-solution trees shifts the focus from “how do we deliver X?” to “which options best help us reach our target outcome?”.

My top tips for opportunity-solution trees are:

• Focus on a single product outcome. Don’t use your tree to map out the full range of ways that your product could drive your business forward, as it will quickly become overwhelming.

• If you’re struggling to identify a single product outcome to focus on, then you’re missing a clear product strategy. Work on that first, at least to the point that you know what outcome is the most valuable to focus on right now. (Here’s a great Product Breaks article on product strategy.)

• Think of your tree as providing a snapshot of the status of your discovery. So annotate it to show where you’ve ruled out options, and to highlight the key data points and useful insights that you’ve learned along the way. But balance that with keeping the tree easy to digest and understand. Link through to documents for more detailed learnings and explanations where necessary.

Choosing a path

2. RICE scoring

Intercom developed RICE to assist with prioritisation decisions. It uses quantifiable measures of Reach, Impact, Confidence, and Effort to calculate a RICE score. When choosing between a list of similar options, it helps to highlight where you can make the biggest impact in the shortest time. I’ve found that the RICE metrics add a useful framing to solution ideas, which help me to prioritise which discovery activities to focus on next.

But my experience has taught me to use RICE with caution. The comfort of the maths can give you a false sense of certainty if used in the wrong way. You should be careful not to use it as a substitute for deeper thinking. Here are my tips for the effective use of RICE.

• Compare apples to apples. Only use RICE to compare options aimed at the same outcome. If not, you aren’t comparing like with like. Only consider RICE when you have decided on the right outcome to strive for.

• Prioritise opportunities before you prioritise solutions. You can’t do a full RICE calculation for opportunities, but you do need to assess which to focus on first. For each opportunity, find a way to evaluate its reach - the percentage of your target market that the opportunity is relevant to - before you look for solutions.

• Don’t immediately dismiss low-confidence ideas. If there’s a possibility of high impact and high reach, do the work to test your assumptions and improve your confidence score. Don’t just switch to a lower-impact idea instead.

3. Confidence meter

RICE relies on an assessment of your confidence in the reach and impact of your idea. But confidence is subjective, so you need a defined scale for it to make sense. Itamar Gilad’s confidence meter provides exactly this. I love it because the scale definitions are objective, while the scoring levels are not linear. It’s designed to strongly incentivise evidence-based decision-making. A small amount of market data is 100x higher confidence than your own self-conviction. And launch data is 10x higher confidence again.

My top tips when using the confidence meter are:

• Make sure you and your team always refer to it for your RICE scoring.

• Continue to refer to the confidence ladder throughout the assumption testing and launch phases of your feature. When planning your release ladder, you should aim to continually de-risk the feature as you make it available to more users.

4. Assumption mapping

Assumption testing is another one of Teresa Torres’s core foundations of product discovery. After you uncover the assumptions behind your ideas, she suggests testing only the riskiest ones. Her assumption map lets you place your assumptions on a 2x2 chart, plotting their relative importance versus their level of evidence (another opportunity to use the confidence meter!). The riskiest assumptions are those with high importance but low evidence.

I love this technique because it encourages you to test only what’s crucial to ensure the idea can succeed. The reduced testing scope allows you to learn and make decisions at a faster pace, instead of building and testing entire solutions.

My top tips for assumption mapping are:

• Write assumptions so they must be true for your idea to be successful. It might not sound important, but it makes it much easier to understand the meaning of proving or disproving your assumptions.

• To visualise your learning, connect your assumption tests to your opportunity-solution tree.

• A simple discovery-focused kanban board can be useful to track your team’s tasks needed to test your assumptions.

Planning ahead

5. GIST Board

Itamar Gilad’s GIST board maps out the Goals, Ideas, Steps, and Tasks you need to take your ideas forward into full-fledged, successful features. I’ve found GIST boards to be both similar and complementary to the opportunity-solution tree.

Opportunity-solution trees are great for moving from a blank page to a structured wealth of ideas to explore. GIST boards can then help to chart your course through that complexity. They can also connect your discovery through to delivery and launch in a way that opportunity-solution trees are less suited to.

It wasn’t immediately clear to me how opportunity-solution trees and GIST boards relate to each other or work together. So after a working with both for a while now, here are my tips for using them together.

• Start with an opportunity-solution tree and use it to guide your ideation and to evaluate which avenues to explore first.

• When you’ve narrowed down the solutions that you’re going to explore further, add them to your GIST board. I map ‘Solutions’ from my opportunity-solution tree map to ‘Ideas’ on my GIST board, and ‘Outcome’ to ‘Goal’.

• Map out the steps needed to build confidence in the solution on your GIST board. I frame the steps as what we need to achieve to take our idea up through the levels of the confidence meter.

• Track the outcome of each assumption test on your opportunity-solution map. Use the resulting increase in confidence to progress through the steps on your GIST board.

Discovery beast: tamed

Product discovery is, by definition, messy. It’s a process of exploration, experimentation and de-risking. So, it’s right that there’s not one clear path forward. But that doesn’t mean it must be chaos. These five techniques have helped me to stay in control. They’ve helped me to:

• ✅ Understand where to focus

• ✅ Chart a course forward

• ✅ Collaborate with ease

• ✅ Keep stakeholders in the loop

This year, a handful of products reminded me what care in product design feels like. Not only did they perform effectively, but they also reshaped my expectations.

As we head into the holiday season, it presents a natural moment to reflect on what has stood out. One of my favourite books A Tribe of Mentors, asks a simple but powerful question about purchases a less than $100 that have meaningfully improved your life. That prompt got me thinking in the year of AI, which products genuinely changed how I think, work and create.

This article provides a nice bookend to our year in articles mirroring our first product breaks article of the year, which you can check out here: Using AI to supercharge your PM powers. These are the 5 products that stayed with me, and the principles they reinforced for the future of product.

Design for Flow, Not Features

Product: Notion

What delighted me about Notion was how little pressure it put on me to get things right upfront. I could open a blank page, start messy, change my mind halfway through, and reshape my thinking without friction. At one point, I realised I could turn a rough paragraph into a toggle or structure it into something more formal without breaking my flow. The product never interrupted me to ask what I was trying to build.

Notion felt like it trusted me. It did not rush me, correct me, or overwhelm me. It let me think in my own shape.

The lesson here is not about capability. It is about flow. Great products optimise for the mental state they want users to be in. They reduce cognitive load and create space for momentum. Productivity becomes a by-product rather than the goal itself.

Takeaway:

Design for the core mental state you want users to experience. Features are secondary.

Trust As a Product Feature

Product: Monzo

Monzo continues to stand out because it consistently finds ways to remove anxiety from money. The 1p 365-day savings challenge delighted me because it turned saving into something approachable and quietly rewarding. Direct investing from my bank account removed the mental barrier between intent and action. My mortgage checker did not just show numbers but offered clear recommendations I could act on. Even merging all my pensions into one place felt less like a financial admin task and more like regaining clarity. At no point did the product make me feel foolish, rushed, or uninformed. It explained itself. It earned confidence over time through consistency.

This is what trust looks like in practice. Not grand gestures, but predictable care in moments that usually cause stress.

Takeaway:

If users do not trust your product, no amount of functionality will compensate. Transparency is foundational.

Lowering the Cost of Creation

Product: Lovable

Lovable delighted me by making starting feel cheap and reversible. I could move from an idea to something tangible in minutes without losing momentum to setup or configuration. There was no sense that I needed to commit to the right structure upfront. I could explore freely, experiment, and change direction without penalty.

That ease invited curiosity. It made play feel acceptable again, which is rare in creation tools. The craft here lies in reducing activation energy. When the cost of getting started is low, people explore more, learn faster, and stay engaged longer.

Takeaway:

The faster users can move from intent to outcome, the more likely they are to explore, learn, and return.

Product Should Learn With You

Product: ChatGPT

ChatGPT delighted me in moments where I did not yet know what I was asking. You don’t need to know the “right” way to use it. You learn by doing. The product adapts to your style, your context, and your level of intent. I could return to a conversation days later and pick up where I left off without re explaining context. As I worked, the responses adapted to my tone, depth, and way of thinking. It helped me clarify ideas that were still half formed.

It did not demand precision upfront. It helped me arrive there. Over time, it stopped feeling like a tool I operated and started feeling like a system I collaborated with. That shift is subtle, but powerful.

Takeaway:

The most powerful products do not just respond to users. They evolve with them.

Respecting User Time

Product: Linear

Linear delighted me through speed and restraint. Actions happened instantly. Keyboard shortcuts felt discoverable rather than hidden. I never felt like the product was asking for more attention than necessary or slowing me down with ceremony.

It was opinionated, constrained, and unapologetic. In doing so, it sent a clear signal. My time mattered.

That respect showed up in every interaction.

Takeaway:

Simplicity and speed are not just UX choices. They are signals of respect.

What These Products Have in Common

None of these products are perfect. What they share is care.

Care in how decisions are made.

Care in what is prioritised.

Care in how users are treated over time.

They serve as a reminder that great product craft is not about novelty or scale alone. It’s about sweating the small stuff, staying intentional, and remembering that every interaction - no matter how minor - shapes how a product is experienced.

These products did not just raise my expectations as a user. They raised the bar for what I am willing to build as a product manager.

Because the best products don’t just function.

They feel considered.

Which products did you use that make your list this year? Share in the comments below. Happy holidays and see you in the New Year!

Back to reality (I think my chances of becoming an F1 driver are probably gone now), and I’m fresh back from kicking off a brand new team with a brand new set of goals to go after. The start of every new team, project, or initiative is one of the most exciting periods for any product manager. Everything is on the table and nothing seems impossible. The roadmap is full of possibility rather than compromise, and teams tend to be full of ideas on how we can make it a reality. What’s more, we’re meeting new people and forming new relationships. The potential feels untapped.

Over the years, I’ve learned that what you do in this opening phase determines far more than your first few months of delivery. It sets the emotional, cultural, strategic, and psychological foundations for everything that follows. Investing time, energy and thought into these early stages can have an exponential effect on the performance of your team over the coming months and beyond.

Start With Energy & Trust

Strong team morale can help you overcome even the most difficult challenges, but it isn’t something that happens by chance. It requires deliberate effort and putting in that effort from the outset can set the tone for working brilliantly together from this point onwards.

Our ultimate goal should be to foster teams that feel:

• Safe to speak up

• Excited about what they’re building

• Clear on how they can contribute

• Respected for what they bring

In the early stages of a new team or initiative, it’s natural for people to be figuring out how decisions are made, whether their opinion is valued, or how each of their colleagues work. Therefore it can be really helpful to lean into these themes during kickoff sessions. Consider:

• Sessions designed to foster psychological safety

• Introductions which eschew the traditional (and, let’s be honest, awkward) activities such as “tell us something interesting about yourself” and lean more towards what makes each individual tick, what’s their preferred communication style, what are their working hours, etc

• Overviews of each individual’s passions and strengths, and what they expect you to come to them for

This is even more critical in the post-pandemic/hybrid world we all now live in. We can no longer rely on these bonds forming organically – we must be intentional and deliberate about it.

Develop a Shared Understanding

High performing teams are rarely content with simply delivering against a set of requirements that are handed to them. The most beautiful Miro board or thorough PRD will never be a substitute for a shared understanding of why we’re doing what we’re doing, how it drives value for customers, and what success looks like for us as a team.

It can be helpful to either share or co-create any of the following (non-exhaustive) list:

• The business’ strategy and any product strategy derived from it

• The overall vision for the product

• Any KPIs or, even better, OKRs associated with the product so that the whole team can understand what outcomes we’re really trying to achieve

A unified team with a passion to innovate and deliver against those strategic goals can emerge through discussing, debating, and iterating on those key artefacts. It’s built when assumptions and complexity are surfaced, not shied away from, and when team members feel free to ask “stupid” questions. Gathering the whole team around these concepts from the start allows for smart trade-offs to be made throughout the product life cycle.

At the start of a project, your real job isn’t to define requirements - it’s to create alignment around intent. What problem are we truly solving? Who are we solving it for? What does success feel like? How will we measure it?

When teams share meaning and purpose, not just deliverables, speed and quality of delivery go up.

The Product Manager as a Cultural Architect

We often think of product managers as prioritisers, problem solvers, or champions of user value. But at the start of any new initiative in particular, you are creating something less tangible, but just as valuable: you are architecting a product culture that will be at the heart of every decision and action going forward.

Before we get into roadmaps, backlogs, or ceremonies, we first need to design the environment in which product thinking can thrive. We’re fostering an environment which encourages:

• A focus on solving the actual problems users are facing

• Customer data being an insight that every team member craves – not just the PM

• Experimentation aimed at validating assumptions

• Genuine belief in, and alignment to the overarching strategy – rather than just paying lip service to it

• Differences in perspective, because they help us achieve better product outcomes

And, crucially, we’re doing all of this before any features are discussed. Before the chequered flag is waved. Before the lights even go out to signal the start of the race.

It is possible to foster this culture later in the product life cycle, and of course a strong team culture and product-alignment is never a “set and forget activity”, but you will likely find yourself part of a more motivated and performant team if you can embed this culture from the off.

In Conclusion

Intentionally designing a strong product culture based on safety, trust and shared alignment, will give you the best possible foundation to deliver the long-term value we are all so passionate about.

Therefore it’s important not to rush it. Take the time that’s required, remove any blockers to your team’s talents and passion, and take pride in the amazing things this enables you to achieve.

“Any organization that designs a system (broadly defined) will produce a design that mirrors the organization’s communication structure.”

Melvin E. Conway, 1968

It’s rarely realistic to rip up your company’s organisational structure and start again before you deploy a piece of code. So, how can you work with suboptimal org shapes and still produce brilliant user experiences?

Spot the warning signs

It’s usually clear when you’re working in a poorly designed organisation that could feel the pain of Conway’s Law. Perhaps you’re set up in ‘functions’ such as IT, Finance and Marketing, and you infrequently work with other functions. Maybe your company has an outsourcing model where they contract different parts of a system (e.g. front-end, integration, APIs) to separate, at times competing, third parties.

It all boils down to a common characteristic: your user and their needs are not at the centre of conversations in your organisation. Instead, you spend more time talking about which team is responsible for what. Teams work towards their own, independent goals, and shared, user-focused outcomes are lost.

We have all experienced the negative impact of Conway’s Law in our everyday lives. It’s why you have to repeatedly give the same personal information to public services run by different government departments. It’s why the online experience of traditional banks used to feel like a series of separate company web pages taped together. It’s the reason for those enraging customer service phone calls where you’re passed round and round between different divisions to have a simple question answered.

Bad for users, bad for business

The result for both business and users can be disastrous.

Firstly, it produces a suboptimal user experience that feels like it’s been taped together at random, or where the user gets frustrated because they’re repeatedly providing the same information to different parts of your service. It risks cost and potential incidents because you build systems that are more complex than they need to be, thanks to multiple, misaligned technical strategies playing out in your product.

It’s also massively inefficient. It takes more time to align teams around outcomes, share information and work through issues when you have more people working to opposing incentives. It’s a huge administrative overhead to coordinate independently moving arms to deliver as one.

This is why successful technology companies set themselves up differently. They want to make it as easy as possible for teams to collaborate on creating great products.

What’s a girl to do?

Most traditional organisations don’t have the time or money to shift to a Spotify squad model before delivering their next digital product. So here are some ways you can combat Conway’s Law without radically changing how your teams are set up.

1. Know your users

Possibly one of the most common tropes in modern product development, but for a reason. Ensuring that everyone involved in your project has a shared understanding of who your users are and the problems you’re solving for them is a fundamental starting point for building good experiences. Ensuring that this view of the user is consistent across siloed teams is key to creating a sense of unity and collaboration, even when org structures don’t support it.

Practical tools: regular user interviews, personas that are easy to reference, shared playback sessions.

2. Set priorities based on user outcomes

Misaligned incentives are the most flammable fuel to the hellfire of siloed teams. Planning and managing work around user outcomes are one way to create shared incentives instead. Rather than ‘back-end deliver xxx and front-end deliver yyy’, the goal becomes ‘a user can check-out and pay’ and the back-end and front-end teams work together to break down what that means. This requires strong collaboration between potentially disparate parties, which leads us to the next point.

Practical tools: OKRs, shared goals and metrics, shared mission memos.

3. Facilitate your way to cross-functional alignment

Collaboration between disciplines is the essence of effective software delivery. Where you have siloed, separate teams, achieving meaningful collaboration is hard. You need to actively make it happen through sessions run by experienced facilitators. Session content can be anything from joint planning to solution design and progress tracking. The key is having an impartial facilitator who can bridge siloes and encourage productive contributions. You also need to make sure the relevant people are present for collaboration sessions, to avoid hand-offs and waste later.

Practical tools: agile ceremonies, especially retrospectives, shared roadmaps, co-design workshops.

4. Appoint a single leadership-level product trio

Product trios (usually made up of a product manager, technical lead and designer) are the decision-making engines of effective teams. If your organisation is building a product through the efforts of multiple disparate teams, it’s essential to place a strong product trio in a leadership position, where they can look across teams to make final calls about prioritisation, technical approach and user experience. A single product, tech and design voice is how you protect against feature bloat and your product feeling like multiple confusing and separate experiences.

Practical tools: three amigos ceremonies, decision logs, shared technical, product and design strategy docs.

5. Promote transparency and trust

Finally, if your org structure doesn’t naturally promote sharing and collaboration, you can achieve it by creating a culture where transparency and trust are the status quo. This means encouraging teams to share their work and plans, even when (especially when) they’re stuck or don’t have confidence in them. It means enforcing a no-blame culture, where failures are opportunities to learn rather than things to hide. Without this openness, it’s very hard to achieve the collaboration required to build great products. An open approach needs to be role-modelled from the top down as well as practiced from the bottom-up.

Practical tools: demos, show and tells, week notes.

Your fate doesn’t have to be your destiny

Conway’s Law is undeniable, but there are ways to combat it. It takes active, persistent and self-scrutinising actions. But it’s worth the effort if you want to build experiences that delight users and achieve results for your business.

I have been thinking a lot about leadership recently, both how to demonstrate it myself and how to foster it in my teams. Here are ten short notes about leadership:

1. The absence of leadership feels like drifting. There is uncertainty that can’t be resolved. Uncertainty isn’t itself a bad thing (it is, naturally, a part of product work, especially in discovery) but it needs to be worked through with confidence.

2. A feeling of drifting has to first be managed in yourself. When you notice it, no matter your role or your level, first ask yourself, what can I do to re-focus my team? Being able to articulate the why behind the work your team is doing is a powerful thing. Write it down in a sentence.

3. Once you have your sentence, and it is clear and to the point, say it to yourself, to your team, to other teams you work closely with, in expanding circles of influence. Make it simple. Reiterate it until it sticks.

4. This itself is part of communicating, early and often, which is a sign of true leadership that is accessible to everyone all the time. Share what you’re doing and why it matters. Share what you’re not doing, and how that frees you to focus on the highest priorities and helps to move the whole team forward. Make it clear how others can do the same, so that the whole team is moving in the same direction.

5. Commit to the people on your team and in your organisation. When you create clarity for others, it shows that you value their time and efforts. You can’t build a product or a platform alone, or, for that matter, a business. Leaders don’t just know this, they make this a central tenet of how they operate by investing in the environments others are operating within (at all levels!). This unlocks individual contributions.

6. Setting good guardrails is key. An autonomous team is not one that can do whatever it wants; it’s one that understands the goals and accepts responsibility for achieving them. That includes making decisions, all the way through to their conclusions, which means accepting consequences. Consequences mean impact: on the team, the business, the customers, and in some cases even more broadly (depending on the scale of your product). Good product teams de-risk their decisions and make calculated bets based on confidence in what they do and do not know. Leaders help teams work through these things by asking good questions, challenging appropriately, helping teams navigate internal processes, and taking ownership.

7. If you have the opportunity to set expectations (with direct reports, or team members) clarify what good leadership looks like. Do this in 1-to-1s by naming the behaviours and skills required. Support these individuals where they need it. Then trust those individuals to lead.

8. It is the job of leaders to set or clarify constraints. Constraints must be managed and adhered to, especially in highly regulated industries or products, but they can also create the right circumstances for creativity to thrive.

9. Good leaders do all of the above. Great leaders do all of the above while recognising and acknowledging those around them who are also leading. Show appreciation when others demonstrate behaviours that signal strong leadership. This creates space for others, which becomes the thing that others can see and copy. Repeatable and rewarded behaviours are what makes a culture.

10. Ultimately, leadership is not a role or a title. It is a skill, and as a skill, it can be learned. Anyone who does not believe themselves to be a leader today, can learn how. It does not need to come with a promotion or the next job. You can do these things today. Imagine what would happen if you did.

Take our survey here.

The AI age has led to more tech start-ups than ever before (.com bubble notwithstanding). In an early-stage start-up, founders have a clear vision and, with that, often take ownership of the product. But as the company starts to scale, they reach a tipping point. Growth demands focus — on building high-performing teams, securing funding, and setting strategy. As the product matures in the market, founder instinct alone is no longer enough. The first product hire becomes essential in shaping a culture that will determine long-term success. Getting that right matters — and this is where trusted fractional product leaders can play a crucial role in reducing risk and setting the foundations for sustainable growth.

If you’re a founder, or work within a VC or PE firm, we’d love your view. If you don’t want to be biased by the rest of the article - jump to our survey here.

Risk 1: Ensure you’ve got the Product Market Fit needed to scale.

When businesses show signs of growth, investors look across every area for evidence of long-term traction. Product-Market Fit (PMF) is one of the clearest signals — and often the key to unlocking funding. Early-stage start-ups can sometimes make it through a seed round, or even Series A, on the promise and storytelling of PMF that hasn’t yet been achieved, but the window to rely on this is short. To succeed long term, a business needs to have identified a real problem worth solving, and solved it in a way people are willing to pay for. Bonus points if that solution is meaningfully unique to the start-up.

For those teams who haven’t found PMF, focus becomes a challenge. Teams juggle an influx of ideas, products and features in an attempt to strike gold (a sure-fire way to burn through precious runway!) If you haven’t found PMF yet, a high-performing product leader who can identify, test and validate the right opportunities is essential.

If you have found it, you’ve completed the 0–1 part of the journey. The next challenge is figuring out what 1–N looks like. Here, high performing Product leaders will work alongside the founder to translate the vision into a clear, scalable product strategy. Sometimes that means doubling down on your core solution for long-term upside; other times, it means using what you’ve learned to expand into adjacent opportunities or new revenue streams. Every business is different, but one constant remains: a trusted product leader who can balance ambition with discipline, and vision with execution, is critical to shaping the company’s next stage of growth.

Risk 2: Ensure you’ve established a good product culture

A founder with a strong product vision is a major asset for getting a start-up off the ground. However, once you’ve reached a scale where the founder can no longer be in the details, the tendency to task engineering teams based on founder insights alone becomes a liability.

At this tipping point, the goal is for the founder to stay opinionated about the right high-level user and business problems to solve, while feeling confident in empowering their teams to explore solutions and uncover new opportunities. A founder who can shift from doing to enabling creates the space for the business to grow.

Establishing a culture where teams have the skills to talk to customers, experiment, and set and measure outcome-oriented metrics is critical. This is how you stay ahead of the challenges that slow larger organisations down - teams aligned to technical architecture instead of user value or trapped in reaction driven cycles that kill innovation.

Building with intention now prevents those problems from emerging later so you don’t have to unpick them or retrofit product fundamentals when you’re preparing for scale or exit.

A product leader who can define strategy, deliver hands-on, and build effective product organisations is rare — and those who can do all three are expensive. A trusted fractional product leader can bridge that gap. They bring experience setting the right foundations, a clear view of what good looks like, and the network to embed strong mid-level talent who can establish best practice from the start. Crucially, they help empower teams early on — building confidence, capability, and the habits that drive high performance as the business scales. The result? A scalable product culture that supports both the founder’s vision and the company’s long-term success.

Risk 3: Ensure you’re hiring the right people

Fractional support isn’t forever — it’s about getting you to the stage where in-house talent can take you to the moon. But that first product hire matters. They set the tone for what you build and how you build it. We believe product people are best hired by good product people, and this is where a fractional product leader, embedded within the start-up, adds real value.

Getting product best practice right is essential, but it’s about more than frameworks and rituals - it’s about fit. A product-first founder needs someone who respects their vision, but with the trust and credibility to challenge it. A sales-first founder needs a product partner who can bring focus and define a clear strategy for which opportunities to pursue — and what to say no to. Every founder needs product people who can embed within teams, deliver impact, and build the habits that drive high performance from the start.

Conclusion....

As product people, these are the patterns we’ve seen emerge time and again in start ups - but we know every growth story is different. If you’re a founder, or work within a VC or PE firm, we’d love to hear your view.

Take our short survey and help us shape our thinking on what it really takes to build high-performing product organisations from the start.

Speaking of taste…

In a conversation with Lenny Rachitsky, Shreyas Doshi (ex-Stripe, Twitter, Google, Yahoo) says that “Taste is the ability to identify what is really good without needing to see its results” (watch the full episode here). It requires zero taste to say, now, that touchscreen phones are better than phones with a physical keyboard. But do you remember that when Apple launched the touchscreen iPhone back in 2007, critics mocked the lack of a physical keyboard, saying it would be slower and impractical compared to a Blackberry, or a Nokia. Do you remember Blackberries and Nokia phones, I wonder?

Similarly, one would say that it’s pretty obvious that short form content is a super engaging format, especially for young people, because all you need to do is look at the companies who are leveraging this to drive engagement among their users to know that this is a valuable offering. It would have looked risky at the time for companies who first took a bet on this back in 2013/14 (short attention spans, mobile-first, user-generated loops?), but it reflected deep intuition about shifting media consumption.

Taste is the ability to identify what is really good without needing to see its results.
Shreyas Doshi

Anyway, I digress. Product sense and having good taste is a topic for another article. 😀

Back to experimentation. I’ve seen in organisations where this becomes a crutch, and a way for teams and/or leaders to avoid making a call. Every team needs to get to the point where they so deeply understand their users, and so deeply understand the competitive space in which they operate, that data is simply directional. It tests a hypothesis enough to de-risk a feature, allowing the team to move forward more confidently.

So instead of asking for more and more tests, leadership is able to Just Decide Already (JDA, you read it here first!)

The comfort of endless testing

Experimentation gives us safety. It signals that a team has taken steps to de-risk a feature or proposition. It makes us feel like we’re doing the “right” thing: staying curious, not just jumping to assumptions.

But too often, it becomes a holding pattern. We keep testing because we’re afraid to be wrong. Because deciding means owning the outcome. Certainly, in places where leaders are quick to blame the team/blame you as the product manager if something goes wrong, I can empathise why you would feel it’s ‘safer’ to completely rely on an experiment effectively telling you what to do.

Truth be told, experiments only help you reduce uncertainty, not remove it completely. The rest is product sense + gut instinct + knowing your users and market + having the courage to just go for it!

The trap of “Do No Harm” experiments

Ah, a ‘do-not-harm’ experiment. This is when leadership wants you to validate a new feature by showing that it doesn’t harm the existing, primary metric. Suddenly, your experiment isn’t truly an experiment anymore. It’s a safety exercise dressed up as ‘science’. Unfortunately – in my view – this is how innovation quietly dies in an organisation.

I get it. No one wants to break something that’s working perfectly well, but that’s how organisations get stuck in doing ‘ok’ versus truly innovating. The best innovations always cannibalise something in the short-run. Take a look at these two examples.

When Apple (I promise, they aren’t sponsoring this post) launched the iPhone, it could have excluded music playback as one of the features in order to protect revenue from iPod sales. And that would have made sense, right? Not Apple. They deliberately integrated a music player with a phone, essentially making their flagship product at the time redundant. They made it obsolete in order to bet on a bigger future. And what a bet it was. In the last 3 months of 2007, iPods made up 42% of Apple total revenue – a sweet $4 billion (ref). Fast forward to today, iPhone sales made up 51% of their total revenue in 2024, around $195 billion (ref).

The same story is seen with Henry Ford (one of our favourite product examples), when he built affordable cars. He didn’t optimise for faster horses, or worry about cannibalising weekly horse usage (WHU? Just go with it 😀). Instead, he completely changed how people travelled.

Now, I’m not suggesting being reckless, and I imagine whoever signed off that decision at Apple would have been pretty nervous at that time. But sometimes you have to trade a little dip in the short term for potentially larger gains in the long term.

It’s about defining what “acceptable harm” looks like, setting smart guardrails, and knowing which metrics truly matter in the long run.

When the numbers say “no” but your gut says “yes”

So I know I said I would cover product sense in another article, but just quickly...

Not every decision will come neatly packaged up in a dashboard, with flashing red arrows saying “launch this feature on the 19^th of November at exactly 11:42am”. Sometimes the quantitative data might look flat, even inconclusive. However, the qualitative signals will be telling a different story.

You’re able to read between the lines when you speak to users.
You can see it in the comments they leave on your App Store reviews (or your competitors...)
You can hear it in the subtleties: “I love this in <insert competitor’s product name here> but I didn’t expect you/your brand were cool enough to have this feature too!” Ouch

Those are the moments when your instinct / product sense matter more than numbers and data.

The best product managers know how to evaluate both – evidence plus instinct. They’ve been building products long enough to know how to feel what the data can’t yet prove. They know when they’ve reduced as much uncertainty as possible that they can make the necessary decision with enough confidence. They also know that if they wait until they have 100% certainty, they will lose time and time again to their competitors.

The real lesson

Data gives you direction, and experiments give you confidence. But ultimately, at some point you’ll need to put your big girl/boy pants on and make a decision. This is what gives your team momentum. This is what sets you apart from the rest.

Besides, what’s the worst that can happen?

A recent MIT report (The State of AI in Business 2025) put it starkly: 95% of organisations fail to generate real value from GenAI. Most never get past pilots and proofs of concept.

But there’s a 5% that do succeed and not just with experiments, but with profitable, scaled outcomes.

So, what’s different about them? And more importantly, what could the rest of us learn?

The ProductBreaks team, working together with clients across many different industries have seen the patterns first-hand. The 5% aren’t doing magic. They’re behaving differently. They adopt certain mindsets that allow them to turn technology into results. And those behaviours are repeatable.

They bridge the gap between process and AI

This is the first and most critical behaviour.

Process owners often don’t know what AI can and cannot do. AI experts rarely understand the messy variation of real-world processes. The result? Misaligned expectations, missed opportunities, and frustration.

Think of it like trying to scale car production. For a one-off, you might hammer aluminium into shape. For a few hundred, you might use fibreglass moulds. For millions, you need pressed steel. But too many teams, when asked to scale, simply hire more people with hammers. Not because they’re foolish, but because they don’t know what else is possible.

The 5% do something different: they put process experts and AI experts in the same room. They encourage honesty about what’s feasible and what isn’t. And they work together to close the gaps.

That collaboration is where scaling starts.

They think in products, not projects

The second behaviour is product thinking.

The 5% begin with the customer problem, the pain point, the unmet need. Because if you’re not solving for customer needs, you’re not relevant.

But they don’t stop there. They look at strategy: does this solution align with where the organisation is going? Does it position us to compete on trust, service, price, or something else that really matters?

And then they ask: Can we do this? Not in a vague way, but with a clear-eyed view of what AI can deliver today, where their own capabilities stand, and where they need to test and learn.

Product thinking is about holding those three truths together: customer; strategy; feasibility and finding the sweet spot in between.

The 5% apply this discipline to AI. They don’t let GenAI become just some R&D. They make it a product.

They build a culture that learns and scales

This is less visible, but it’s decisive. Culture is where you see the real differences between the 95 and the 5.

In the 5%:

• Failure is data, not a career risk. They place small bets, learn quickly, and scale the ones that work.

• Governance is an enabler, not a blocker. Bias, security, compliance, thought through from the start, so teams move faster.

• Data foundations are ready. Clean, integrated, governed. Without it, AI goes nowhere.

• Tacit knowledge becomes explicit. One client’s staff could always recommend the perfect product in person, but their website consistently got it wrong. We fixed that by capturing what staff knew and turning it into data AI could use.

• Change isn’t bolted on. Solutions are embedded into workflows, so people trust and adopt them.

These are cultural muscles. And the 5% keep them fit.

They focus on outcomes, not outputs

Finally, the 5% are obsessed with outcomes.

It’s tempting to measure what’s easy: more clicks, more emails, more activity. But activity is not value.

The 5% look for impact:

• Time saved

• Costs avoided

• Revenue generated

• Customer experience uplifted

For one client in entertainment, the real measure wasn’t clicks. It was TVs sold as a leading outcome, and viewing time uplifted as a trailing one.

From the very first thin slice, the 5% measure ROI. They don’t wait for a mythical big bang.

The behaviours that matter

If there’s a thread running through all of this, it’s that the 5% behave differently. They:

• Bridge the process ↔ AI gap with cross-functional honesty.

• Apply product thinking to balance customer, strategy, and feasibility.

• Build cultural muscles that make learning and scaling natural.

• Stay focused on outcomes, not outputs.

These aren’t checklists. They’re ways of leading. Ways of thinking. Ways of showing up as an organisation.

The behaviours of the 5% aren’t complicated but they are deliberate. They build trust between experts, discipline into decisions, and resilience into culture. Above all, they measure what matters.

The organisations in the 5% don’t have all the answers. What they do have is a way of showing up: curious, pragmatic, and prepared to change. That is what keeps them ahead.

The real question isn’t can you use GenAI. It’s whether you’re willing, as a leader, to adopt the behaviours that turn potential into profit.

The ProductBreaks team are helping organisations across sectors do exactly that, delivering outcomes, not outputs, and making GenAI profitable in practice.

In previous articles, I outlined:

• The basic principles of Systems Thinking and how it can be used in Product Management.

• How applying Soft Systems Methodology to can help you understand complex socially rooted problems.

• How applying Systems Dynamics can help you understand complex situations and causality.

Critical Systems Heuristics (CSH)

Critical Systems Heuristics (CSH) was developed by Werner Ulrich in 1983. Ulrich observed that traditional systems design often assumed objectivity, overlooking the reality that every decision is shaped by boundaries, values, and power dynamics.

CSH was created to expose these hidden assumptions by asking structured “boundary questions” that highlight who benefits, who decides, whose knowledge counts and who is excluded. Rooted in critical social theory, its purpose is to make decision-making more transparent, inclusive, and accountable, particularly in complex environments where multiple stakeholders are affected.

In Product Management, CSH can be applied by regularly using its boundary questions to check whether the product’s vision, backlog, and metrics truly reflect the needs of all affected users, not just the most powerful stakeholders.

For example, Product Managers can use it within:

• Product discovery – to surface whose needs are prioritised vs overlooked.

• Roadmap shaping – to test if long-term goals reflect only dominant stakeholders or also marginalised users.

• Backlog refinement – to validate whether stories serve broader citizen/user groups, not just policy or commercial pressure.

• Retrospectives – to periodically revisit boundaries (“who’s missing from this conversation?”).

Applying CSH

There are 12 boundary questions, which are grouped into four categories.

• Motivation (why the system/product exists)

• Control (who has power and resources)

• Knowledge (what counts as valid knowledge)

• Legitimacy (who is affected and represented)

The questions within each category are designed to:

• Expose hidden assumptions about beneficiaries, decision-makers, and legitimacy.

• Challenge boundary choices (who/what is in or out of scope).

• Support reflection on ethical, social, and systemic impacts of product decisions.

• Encourage reframing so that design and delivery include those often overlooked.

Questions

The 12 boundary questions.

Motivation

1. Who is (ought to be) the intended beneficiary of the system? Clarifies whose needs and interests are prioritised.

2. What is (ought to be) the purpose of the system? Makes explicit the claimed purpose versus hidden agendas.

3. What is (ought to be) the measure of improvement or success? Surfaces which metrics matter, and whether they serve all stakeholders.

Control

1. Who is (ought to be) the decision-maker? Identifies who holds authority and whether this is legitimate.

2. What resources are (ought to be) controlled by the decision-maker? Shows which resources or levers are available and who commands them.

3. What conditions are (ought to be) outside the decision-maker’s control? Acknowledges external constraints and limits of influence.

Knowledge

1. Who is (ought to be) considered an expert? Highlights which voices are valued for expertise.

2. What expertise is (ought to be) consulted, and why? Surfaces bias in which knowledge is prioritised.

3. What or who is (ought to be) assumed as the source of knowledge? Exposes reliance on particular data, models, or narratives.

Legitimacy

1. Who is (ought to be) affected but not involved? Brings forward marginalised or excluded stakeholders.

2. Who is (ought to be) the guarantor of those affected? Identifies who speaks or advocates for the excluded.

3. What worldview is (ought to be) assumed and legitimised? Surfaces underlying cultural, political, or ethical assumptions shaping the system.

Process

The process to apply CSH within Product Management, you can use the following steps:

1. Frame the product context: Define the service or feature under consideration and clarify the decision or problem you want to explore.

2. Engage stakeholders: Involve a diverse mix of voices from delivery teams, leadership, users, and affected groups so that multiple perspectives are represented.

3. Work through the 12 boundary questions: Organise a workshop or structured exercise to capture both explicit and implicit assumptions.

4. Surface assumptions and tension: Compare different stakeholder responses and identify contradictions or gaps.

5. Analyse implications: Reflect on how current boundary choices shape fairness, inclusion, and outcomes, pay attention to who is excluded or privileged.

6. Reframe product decisions: Adjust the vision, roadmap, backlog, or success metrics to address gaps and improve inclusivity.

7. Document and communicate findings: Use a concise method to share the findings with stakeholders for transparency.

8. Revisit: Review at key milestones so that boundary critique becomes a continuous improvement practice rather than a one-off exercise.

Example

To show an example, we’ll use a fictional scenario, and we’ll apply the first 5 steps, assuming engaged and diverse stakeholders from step 2.

Product context

A government department is developing a digital portal for managing housing benefit applications. The aim is to reduce processing times and improve access for citizens, but there are competing pressures around efficiency, cost, and inclusivity.

12 boundary questions

Motivation

1. Beneficiaries: Policy leads say ‘all claimants’, delivery team narrows it to ‘digitally literate claimants’, housing charities argue the main beneficiaries must be ‘vulnerable groups least able to access digital services’.

2. Purpose: Senior leaders stress ‘cost savings and efficiency’, product team emphasises ‘improved user experience’, charities prioritise ‘ensuring no one is excluded from accessing entitlements’.

3. Success criteria: Finance teams cite ‘reduced processing cost per application’, user researchers propose ‘reduced failure demand and higher satisfaction scores’, caseworkers value ‘fewer manual interventions’.

Control

1. Decision-maker: Department’s digital director holds authority, but delivery teams highlight strong influence from Treasury funding conditions.

2. Resources controlled: IT budget and developers are department-controlled; housing charities highlight that ‘frontline support funding’ is not.

3. Constraints: Legislative rules around benefits, cybersecurity standards, and procurement frameworks are acknowledged as outside local control.

Knowledge

1. Expertise recognised: Policy and technical SMEs are seen as experts; user researchers argue ‘lived experience of claimants’ should also count.

2. Knowledge sources: Quantitative transaction data is prioritised, researchers and charities push for ‘ethnographic insights and community input’.

3. Assumptions about knowledge: Leaders assume ‘data dashboards give a complete picture’, advocates argue this misses hidden barriers such as low digital literacy.

Legitimacy

1. Affected but not involved: Vulnerable claimants with no internet access, people with language barriers, and those with disabilities are identified.

2. Guarantors of the affected: Housing charities and local councils see themselves as advocates but note they’re rarely included in decision-making.

3. Underlying worldview: Department frames benefits as a transactional service to be streamlined; advocates argue it should be seen as ‘a right and a lifeline for vulnerable citizens’.

Assumptions and tensions

• Efficiency vs. equity: The department assumes digital automation automatically equals fairness, while advocates argue it risks excluding those most in need.

• Knowledge hierarchy: Quantitative data is treated as “hard evidence,” while qualitative lived experience is sidelined, creating tension about what counts as valid input.

• Representation gap: Vulnerable groups are acknowledged as affected but structurally excluded from design conversations.

• Purpose clash: Cost savings are the official purpose, but teams working directly with claimants see citizen wellbeing as equally important.

Implications

If boundaries remain as currently assumed, the product risks becoming a ‘digital barrier’ rather than an enabler, cutting processing costs but worsening access for digitally excluded claimants. Failure demand may rise (more calls, complaints, appeals), undermining efficiency claims. Ignoring lived experience creates reputational and ethical risks, the service could be criticised as discriminatory or non-compliant with accessibility standards.

Reframing assumptions by broadening “beneficiaries” to explicitly include vulnerable groups, recognising lived experience as valid expertise, and adjusting success metrics to include fairness would align the product with both efficiency and inclusivity, strengthening legitimacy and reducing long-term risks.

Insights

This insight teaches us that assumptions about efficiency, data, and control can unintentionally create exclusion if they aren’t challenged. By surfacing hidden boundaries, such as who counts as a beneficiary, what evidence is considered valid, and who is absent from decision-making we see that a product designed to “streamline” may in fact widen inequality.

It also shows that different stakeholders hold fundamentally different views of success: senior leaders may prioritise cost savings, while frontline staff and advocates prioritise citizen wellbeing. Unless these tensions are made explicit, the product risks drifting towards narrow objectives that undermine legitimacy.

Most importantly, the exercise highlights that inclusion is not automatic in digital transformation, it must be deliberately designed into goals, metrics, and governance. Applying CSH early reveals these gaps, giving Product Managers the chance to reframe decisions and balance efficiency with fairness, leading to services that are both effective and equitable.

Conclusion

Critical Systems Heuristics provides product managers with a structured way to make hidden boundaries, assumptions, and power dynamics visible in complex delivery contexts. Its process, framing the product, engaging diverse stakeholders, working through boundary questions, surfacing tensions, analysing implications, and reframing decisions ensures that decision-making is not dominated by narrow metrics or single perspectives.

The housing-benefit portal example illustrates how CSH exposes a clash between efficiency and equity: leaders valued cost savings, while advocates stressed inclusivity for vulnerable citizens. By surfacing assumptions (that digital equals fair access, or that dashboards tell the whole story), the process revealed risks of exclusion, reputational harm, and failure demand. Analysing these implications showed that reframing beneficiaries, success measures, and sources of expertise could align efficiency with fairness.

In short, CSH equips product teams to build not just effective services, but also legitimate and equitable ones. It reminds us that technology is never neutral, the boundaries we set determine who benefits, who is excluded, and how public value is defined.

Systems Thinking for Product Managers (to be continued..)

This is the fourth article in a series I’ll release within Product Breaks, focusing on Systems Thinking methodologies and Product Management. Further articles will aim to cover the topics:

• Viable System Model (VSM)

• Strategic Options Development and Analysis (SODA)