Subscribe: Apple Podcasts | Spotify | TuneIn

Experiments to inform product decisions

Watch on YouTube

TLDR

Stefan Thomke, professor at Harvard Business School, and author of The Scientific Mindset: How Thinking Scientifically Drives Better Business Decisions, shares five elements of a scientific mindset, four types of uncertainty in innovation, and three tests of a good hypothesis. Learn why good product decisions are rooted in testing more and assuming less.

Introduction

When Stefan Thomke studied almost 22,000 online experiments (mostly A/B tests) run by more than 1,300 companies, only about one in five changes made a measurable difference. Half of the changes made things worse. If experienced teams can’t predict the effect of a small change, how confident should any of us be about a creating a new product?

Stefan is the William Barclay Harding Professor of Business Administration at Harvard Business School, where he has studied experimentation and innovation for more than three decades. He started as an engineer, learning experimental design to solve a production problem in a semiconductor plant. He is the author of Experimentation Matters and Experimentation Works. His new book, The Scientific Mindset: How Thinking Scientifically Drives Better Business Decisions, completes the trilogy.

Stefan was on the show in 2020 to explain how to run product experiments. This time we look at the thinking that decides which experiments are worth running and what you do with the answers. You’ll learn how to find the assumptions hiding in a new product idea using four kinds of uncertainty, how to turn an assumption into a strong hypothesis, and why “customers want this feature” isn’t yet a hypothesis.

Summary of Concepts Discussed for Product Managers

Five Elements of a Scientific Mindset
Stefan opens with a question: What did you do the last time you didn’t know what to do? He quotes Richard Feynman: “The first principle is that you must not fool yourself, and you’re the easiest person to fool.” The human brain is a belief engine. We see connections between unrelated things and confuse correlation with causation. To overcome this, we need to think scientifically about decisions.

Stefan identifies five elements of a scientific mindset that apply to business decisions:

1. Be a knowledgeable skeptic.
2. Articulate testable hypotheses.
3. Produce hard evidence.
4. Probe cause and effect.
5. Create an experimentation culture, a shared scientific mindset.

Skeptics vs Denialists
A skeptic is not a denialist. A skeptic is neutral. They look at the evidence with an open mind and want the truth. A denialist starts with a conclusion, picks evidence that supports it, and dismisses evidence that contradicts it. Stefan calls this running the scientific method in reverse. It’s easy to fall in love with a conclusion and only question the evidence that disagrees with us. Product managers do this when they run customer research to validate a solution instead of finding unmet needs. Instead, fall in love with the customer’s problem, not your solution.

Test More and Assume Less
The greatest danger zone is when you are most confident in your assumptions and under pressure to act. That is the time to test your assumptions. Stefan explains that information is most valuable when it is most surprising. Effective leaders don’t trust their gut. They test their gut.

Four Types of Uncertainty in Innovation
Stefan sees four types of uncertainty in innovation:

• R&D uncertainty: Does the product, service, technology, or process work as intended?
• Scale-up uncertainty: Can we produce it at high volume, high quality, and reasonable cost?
• Market or customer uncertainty: Are customers interested, will they pay, and how should we price it?
• Investment uncertainty: How much money and how many resources should we commit?

Why Experience and Data Aren’t Enough
Stefan explains that while experience and data are both valuable, they have limits. Relying on experience and data too heavily can prevent you from testing your assumptions. It is easy to mistake correlation for causality. Use experiments in addition to experience and data, because experiments can prove you wrong.

Three Tests of a Good Hypothesis
Stefan describes three tests for a hypothesis:

• It has measurable variables.
• It is simple.
• It can be proven wrong.

Wolfgang Pauli would reply to a bad hypothesis with “This is not only not right. It’s not even wrong.” A hypothesis that cannot be falsified is worse than one that is simply incorrect.

Stefan gives an example of a good hypothesis, which Kohl’s tested: “Opening our stores one hour later will have an insignificant impact on store revenue.” This is a measurable hypothesis they could test in a subset of stores. On the other hand, “We can take our brand upmarket,” is a bad hypothesis since it is not measurable.

Designing Fast, Frugal Experiments
Experiments should be:

• reversible
• inexpensive, at least until they lead to large commitments
• at an appropriate level of rigor for how much uncertainty is acceptable. Some product decisions require maximum rigor. Many do not.

Pay attention to customers’ actions, not only words.

Useful Links

• Check out Stefan’s new book, The Scientific Mindset: How Thinking Scientifically Drives Better Business Decisions
• Connect with Stefan on LinkedIn
• 281: The right way to use experiments to create better products more quickly – with Stefan Thomke

Innovation Quote

“Science is much more than a body of knowledge. It is a way of thinking. This is central to its success. Science invites us to let the facts in even when they don’t conform to our preconceptions.” – Carl Sagan

Application Questions

1. Think about the last time you didn’t know what to do on a product decision. Did you rely on experts, data, your gut, or an experiment?
2. Where might your team be running the scientific method in reverse, collecting evidence to support a solution it has already chosen?
3. Which of the four uncertainties is largest for your current product, and what are you doing to reduce it?
4. Pick one assumption your team holds about customers. How would you rewrite it as a hypothesis with measurable variables that someone could prove wrong?
5. What reversible, low-cost experiment could you run in the short term that measures customer behavior rather than customer opinion?

Bio

Stefan Thomke is a leading authority on the management of innovation and the William Barclay Harding Professor of Business Administration at Harvard Business School. A management educator, speaker, and advisor to global leaders, his pioneering research has inspired organizations across industries to incorporate experimentation and scientific thinking into how they make decisions.

Stefan has published more than one hundred articles, cases, and notes, and is the author of four books: “Experimentation Matters” (2003), “Managing Product and Service Development” (2007), “Experimentation Works” (2020), and “The Scientific Mindset” (2026). “Experimentation Works” was recognized by Forbes as one of the top technology books of 2020, as a top business book of 2020 by Inc. and CEO magazines, and became a Top Shelf Pick by Strategy & Business. 

Stefan received his PhD in electrical engineering and management from the Massachusetts Institute of Technology (MIT). He grew up in Calw, a town in southern Germany’s Black Forest, and now lives outside Boston.

Thanks!

Thank you for taking the journey to product mastery and learning with me from the successes and failures of product innovators, managers, and developers. If you enjoyed the discussion, help out a fellow product manager by sharing it using the social media buttons you see below.