Going Beyond Quantity for Measuring Product Discovery Adoption

Running more experiments is not always an indicator of quality –– in some cases, it is, in fact, only an indicator of quantity. Expressions of volume can’t necessarily tell you if Product Discovery is being successfully adopted or resulting in core outcomes like making better decisions about the solutions you invest time in. It is entirely possible for employees to plug-and-play experiment quantity just to meet goals set by managers. 

Rather, you need to look for a mix of signals that push beyond quantity and into quality to understand how “successful” Product Discovery should look in your organization. 

How to Think About Quality

How do you think beyond performance metrics like the sheer number of experiments to take quality into account? You need to look for indicators of quality. That means looking not just at outcomes but also at processes and workflows.

Some examples of quality indicators could be:

• The number of non-product or UX participants in ideation workshops, interview observations, or Discovery check-ins
• Cycle times for the collection of first-hand signals from users or customers
• The share of features in Delivery that are based on quant/qual testing of critical usability/desirability/feasibility assumptions owned by the Product team

Just like the definition of OKRs benefits from holistic Key Results for measuring your Objective, measuring success for changed ways of working on Discovery needs more than performance indicators.

When thinking about quality metrics, take into account:

• The whole span of Product Discovery to find green light indicators at different parts of the process, not just at the end
• Accounting for behaviors that are key parts of your process which may not be captured by traditional metrics, like how much your teams participated in informal knowledge-sharing practices, like lunch Q and A sessions
• Changes in the behavior of your team or customers that may not be linked or measured for inclusion in other metrics, such as qualitative differences in the problems direct reports surface to managers with regards to Discovery

Pitfalls of Continuing the Quantity-Only Approach

When teams embrace Product Discovery, it’s often due to the fatigue of being on the hamster wheel of building more features that aren’t needed and don’t contribute to business goals. But putting the number of experiments conducted front and center only swaps out user stories with “number of <insert experiment technique here>”.

The volume of experiments conducted tells you that a team worked on tasks other than “building.” But it doesn’t tell you much about whether the team chose the proper experiment techniques for the challenge, executed them in a way that generated valuable insights, and even if they (in-)validated assumptions around the most promising idea–Or, how adaptable their Discovery approach is as a whole.

While you might argue that every experiment (if set up correctly) generates some insight and COULD help the team succeed, this often feels like sandbagging and leads to increasing inefficiencies.

Committing to quality metrics ensures that your team actually does stop spinning its wheels for nothing.

They should give you a window into whether or not your processes are adequate or still suffering from underlying problems that will eventually come to haunt you.

When to Implement Quality Metrics

If your team’s Product Discovery is not where you want it to be  –– you know that Discovery processes don’t help teams reduce uncertainty and you’re trying to change them –– when should you start measuring quality? Do you have to wait until you’ve arrived at an Adapted Product Discovery state in order to hone in on the quality metrics that will work for your team?

The short answer is no, you should not wait. There is no such thing as the perfect time to improve Product Discovery, and since your company is doing some form of Discovery or another already, waiting for optimal conditions is ultimately procrastination. Beyond that, three very good reasons not to wait too long are as follows.

First, introducing quality metrics earlier gives your team a clearer understanding of how Product Discovery processes work together as a whole, and shows them a new way to think about their own roles outside of hitting numeric targets. When you are working on bettering a Product Discovery approach, pairing any new methods, experiments or workflows with better quality indicators gives employees guide rails as they adapt to different processes. 

Second, you may not land on the right quality metrics right away. As with any other indicators, these can take time to hone and get right. Engaging with these metrics alongside your team’s progress gives you more runway to experiment and more feedback to fine-tune. Waiting until every other part of your process seems to be working well before turning to combining quantitative and qualitative analysis all but ensures improving Discovery takes as long as possible.

Third, and finally, because finding these new ways of measuring improvement and success in Discovery will help you hone in on all the other parts of the Product Discovery and its adoption that you target for upgrading. Trying to measure big changes with old metrics –– metrics you know don’t give you the full picture –– is doing things without the clarity you need to drive towards success.

Another Piece of the Puzzle

Both quality and quantity can be important in measuring Product Discovery adoption. But alone neither will give you the full picture of the efficacy of your teams or the success of your efforts to better Discovery at your company. Rather, you should think of Product Discovery adoption the same way as you would think about a product –– something with The Stepstone Group used to better their adoption thinking. Because quality and quantity are two sides of the same coin: they can help you determine not only how well your Discovery is going in the short-term, but also how sustainable the changes you make are for the long run.