How to Set Success Metrics for Lean Experiments

Set your success metric before you run the experiment, not after you see the results. Pick one decision metric that maps directly to the assumption you're testing, set a numeric pass/fail threshold, size the sample, put a deadline on it, and commit to all four in writing before the first visitor arrives. Everything else — the dashboard, the debrief, the "did it work" conversation — follows from that one decision made in advance.

Quick Answer: Before you launch any lean experiment, write down the single metric that will decide the outcome, the number it has to clear to count as a pass, the sample size you need, and the deadline for collecting it. Sign your name to it. If you can't state the threshold in one sentence before you start, you're not ready to run the test.

Most experiment post-mortems don't fail because the data was bad. They fail because nobody agreed, ahead of time, on what a good result would look like — so the team argues about the number after the fact instead of reading it against a standard. The rest of this guide walks through how to fix that, with a scored worked example and a reference table you can adapt to your own experiment.

Prerequisites: One Assumption, One Named Decision

You need two things before you can set a meaningful metric: a specific, falsifiable assumption, and a named person who has agreed in advance to act on the result.

"People want this" isn't testable — it's a mood, not an assumption. A testable assumption names the who, the what, and the condition: "Freelance bookkeepers who manage eight or more clients will pay $39/month for automatic transaction categorization that beats their current manual spreadsheet process." That's specific enough that one metric can plausibly confirm or kill it.

The named decision matters just as much as the assumption's precision. If nobody is bound to act on the number — if a founder can eyeball a marginal result and proceed anyway — the metric is decoration, not decision-making. Before you launch, ask: who signs off on interpreting this result, and what will they actually do differently if the number comes back a no? If the honest answer is "probably nothing," you don't have an experiment; you have a confirmation exercise with extra steps.

If you haven't yet decided which kind of test fits your assumption, start with the Lean Experiment Playbook, which covers smoke tests, landing pages, concierge MVPs, and Wizard of Oz prototypes before you get to the metrics stage covered here.

Choose the Decision Metric

Pick exactly one metric that measures the behavior your assumption depends on — not every number you could plausibly track.

Every experiment throws off a dashboard full of numbers: page views, time on page, click-through rate, email opens, survey responses, social shares. Only one of those is your decision metric — the number that, crossing a line you set in advance, actually ends the debate. Everything else is context you can look at afterward, but it doesn't get a vote.

Alistair Croll and Benjamin Yoskovitz call this your One Metric That Matters in Lean Analytics — the single number that, at this stage of the business, outranks everything else on the dashboard combined. Choosing it well comes down to matching the metric to the kind of commitment your assumption actually requires:

• If the assumption is about interest, the decision metric should be an opt-in action — an email signup, a waitlist join, a "notify me" click — rather than passive attention like page views or scroll depth.
• If the assumption is about willingness to pay, the decision metric has to involve real money or a real commitment: a live checkout, a refundable deposit, a signed pre-order. A survey question about hypothetical willingness to pay doesn't qualify.
• If the assumption is about workflow fit, the decision metric is completed usage — a concierge task finished, a Wizard-of-Oz request fulfilled and actually used — rather than expressed enthusiasm in a feedback call.

The test for a well-chosen metric: would a skeptical cofounder accept this number as real evidence, or would they say "that's not really the behavior we care about"? If you can't answer confidently, you've picked a metric that's easy to move rather than one that's hard to fake.

Set the Threshold

Write down the specific number — a percentage, a count, or a ratio — that separates a pass from a fail, before you see a single data point.

A threshold isn't "as many as possible" or "enough to feel encouraged." It's a line: "12% of landing-page visitors join the waitlist" or "at least 5 of the first 15 concierge customers pay to continue into a second cycle." Anchor the number to something you can defend — a past experiment you ran yourself, a comparable benchmark you've personally verified, or simply the smallest result that would justify the next round of work and spend.

If you have no reference point at all, reverse-engineer the threshold from what would actually need to be true for the next stage of the business to make sense, rather than picking a number that just feels optimistic. For a landing-page test specifically, How to Design a Landing Page Experiment That Proves Demand walks through the traffic math for turning a target signup count into a defensible conversion threshold.

David Bland and Alexander Osterwalder make the same case in Testing Business Ideas: decide your pass/fail criteria before you run the test, not after. Interpreting evidence after you've already seen it is a fundamentally different — and much less honest — exercise than judging it against a standard you committed to in advance.

Define Sample Size and a Time Box

Decide, before you launch, how many observations you need and how many days you'll collect them for — and stop at whichever limit you hit first.

At early-stage volume you rarely need a formal statistical power calculation. What you need is enough observations that one enthusiastic friend, one unlucky day, or one referral from a Slack community doesn't single-handedly swing the result. Many practitioners treat something in the range of 20-30 independent attempts per variant as a rough floor for a simple pass/fail test — not because a formula demands that exact number, but because below it, a single outlier can flip the outcome on its own. Adjust upward for a smaller expected effect size, and don't treat this as a substitute for real statistical rigor once you're operating at meaningful scale.

The time box matters as much as the sample floor. Pick a calendar deadline tied to the cost of waiting — how much runway or opportunity the delay actually consumes — not to when the results start looking good. Two weeks is a common default for a fast, cheap test; a concierge MVP with a real service cycle might need a full billing period. Whatever you choose, write it down before you start, because the moment a borderline result tempts you to "just run it a bit longer," you've already left the sample-size discussion and entered the mistakes section below.

Pre-Commit in Writing

Put the metric, threshold, sample size, and deadline in one dated document before the experiment starts, and share it with someone who will hold you to it.

This is the step most founders skip because it feels like paperwork, but it's the one that actually turns a metric into a decision-making tool instead of a number you'll rationalize later. Without a dated commitment, hindsight bias takes over quietly: a 9% signup rate looks fine if you privately expected 8%, and disappointing if you expected 12% — even though it's the identical 9% either way. Writing "threshold: 10%" on a specific date, before you have a result to react to, is what keeps the eventual verdict honest.

Share the document with a cofounder, an advisor, or even a dated public note to yourself — anyone positioned to call out a post-hoc excuse when the result comes in under the line.

Threshold Reference Table by Experiment Family

Different experiment types produce different strength of evidence, so the decision metric — and what counts as a credible starting threshold — should shift with the experiment family rather than staying fixed. The ranges below are common starting points founders use to set their own threshold, not benchmarks pulled from any single study; recalibrate against your own funnel, audience, and prior experiments before you commit to a number.

*Starting ranges to calibrate against your own prior results — treat them as a reasoning aid, not a universal pass mark.

In practice, the table is less about the exact numbers and more about the pattern: the further your decision metric sits from a real commitment (money, signed usage, a completed transaction), the more skeptically you should read a strong-looking result.

Worked Example: A Landing Page Test With a Clean Go/No-Go

A two-person team is testing whether solo bookkeepers will pay for automatic bank-feed categorization built for messy, multi-client work. Their assumption: solo bookkeepers managing eight or more clients will pay $39/month for categorization that beats their current manual process.

Here's how they set it up, using the steps above:

• Decision metric: percentage of qualified visitors (defined as spending at least 20 seconds on the page) who complete a $10 refundable deposit.
• Threshold: 6% of qualified visitors.
• Sample size: a floor of 400 qualified visitors, sized generously because they expected a small effect from a cold-traffic audience.
• Time box: 3 weeks, funded by a fixed ad budget so the deadline wasn't tied to how the numbers were trending.
• Pre-commitment: written into a shared doc, dated before the first ad ran, co-signed by both founders.

After three weeks: 460 qualified visitors, 31 completed deposits. That's 31 ÷ 460 = 6.7% — above the pre-set 6% threshold. The team also logged secondary numbers along the way — average time on page, which referral channel converted best — but those were context, not the decision. Nobody argued about whether 6.7% "really" counted, because the line had already been drawn at 6% three weeks earlier. The team proceeded: deposits converted into product credit, and the depositors became the first cohort for the MVP build.

Worth noting what almost happened: an earlier internal draft of the plan had one founder wanting to round the threshold down to "roughly 5%, ballpark." Because they'd both signed the written 6% figure, that conversation resolved in about two minutes instead of becoming a week-long debate about what the result "really" meant. That's the entire value of pre-commitment in one sentence.

Mistakes: Moving Goalposts and Proxy Metrics

Two failure modes quietly gut an otherwise well-designed experiment: lowering the bar after seeing a disappointing number, and swapping in a metric that's easier to hit but doesn't actually indicate the assumption is true.

Moving the goalposts

Say the same team's landing page had landed at 4.8% instead of 6.7%. The tempting move is rationalization: "the ad targeting was probably off, let's run it two more weeks," or "4.8% is still meaningfully above zero, that's directionally good." Neither is analysis — both are negotiation with your own result after the fact.

The fix isn't willpower, it's the mechanism from earlier in this guide: the written, dated pre-commitment. Changing the threshold after seeing the number means literally crossing out a number you both signed — friction most people won't create for themselves, which is exactly the point.

Proxy metrics

The second failure is subtler: swapping "would you pay" survey intent for actual deposits, counting email opens instead of click-throughs, or logging "interested" replies on a sales call instead of signed letters of intent. Proxy metrics feel like progress because they're almost always higher than the real number — and that's precisely the mechanism that makes them dangerous. A metric that flatters you by design isn't measuring the thing you set out to test.

This same pattern — reaching for a friendlier metric to avoid a hard call — drives a lot of the false "double down" decisions covered in When to Pivot vs. Double Down. Teams rarely misread genuinely ambiguous data. Far more often, they quietly redefine what counts as a good result until the data stops being ambiguous.

Key Takeaways

• Pick one decision metric that maps to the actual behavior your assumption depends on, not the easiest number on the dashboard to move.
• Set a numeric threshold before you look at any data — a pass/fail line you could state to a stranger in one sentence.
• Size the sample and time-box the test in advance, then stop at whichever limit hits first — never when the results happen to start looking good.
• Pre-commit in writing, dated and shared with someone who'll hold you to it. This single step is what defeats hindsight bias.
• Match the metric to the experiment family — opt-ins for interest, real money for willingness to pay, completed usage for workflow fit.
• Treat goalpost-moving and proxy-metric swapping as the same failure wearing two disguises — both let you avoid a decision you're not ready to make.

Frequently Asked Questions

How many decision metrics should one lean experiment have?

One. A single decision metric with a pre-set threshold is what makes the result unambiguous. Track secondary numbers for context if you want, but don't let them override the metric you committed to before the test began.

What if my experiment result lands right at the threshold?

Treat a near-miss as a no unless you built a buffer zone into your pre-commitment. A result within a point or two of the line is often just noise at small sample sizes — the fix is a wider margin next time, not a post-hoc argument for why this particular near-miss should count.

Can I change the threshold after seeing partial results?

No. That's the moving-goalposts failure this guide is built around. If you have a genuine reason to think the original threshold was set wrong — not just disappointing — stop the experiment, document why in writing, and restart with a new pre-committed number rather than adjusting mid-run.

Do I need formal statistical significance testing for a small startup experiment?

Usually not in the strict sense. Most early lean experiments run at sample sizes too small for meaningful significance testing. What matters more at this stage is a sample large enough that one outlier can't flip the result, combined with a threshold and time box you set before you started collecting data.