Judge the Smoke Test and Cut the MVP Scope

Goal

Judge the smoke-test results with the Wilson interval (per channel, overall, and stranger channels only), calculate the sample needed for on-hold channels, then cut the MVP scope within budget using RICE and decide "whether to build, whether to test more, and what".

Why it matters

A hypothesis can be tested only if you write it as a number that can be wrong, and a test result can be read only if you account even for who took part. The high conversion rate of the acquaintance channel and the high score of an easy-to-build feature are both the most common optical illusions.

Materials

• /opt/fixtures/founder/mvp/smoke.csv — channel,visitors,signups,preorders (newsletter_friends is acquaintances and subscribers; the rest are stranger visitors)
• /opt/fixtures/founder/mvp/hypothesis.json — statement, metric, threshold, budget_person_months
• /opt/fixtures/founder/mvp/features.csv — feature,reach,impact,confidence_pct,effort_pm

Definitions

• Preorder rate = preorders ÷ visitors, signup rate = signups ÷ visitors.
• Wilson 95% interval: the same formula as the previous module (fv-problem), with z = NormalDist().inv_cdf(0.975).
• Verdict: lower bound ≥ threshold → "pass", upper bound < threshold → "fail", otherwise "insufficient".
• n_needed(p, threshold): the first n, trying n = 1, 2, … 200000 in order, for which the interval of k = round(p·n) falls outside the threshold (lower bound ≥ or upper bound <); None if there is none.
• RICE = reach × impact × (confidence_pct ÷ 100) ÷ effort_pm.
• Scope: going through features in descending RICE (by feature name on ties), include one if the cumulative effort_pm stays at or under the budget, and if it overflows, skip it and look at the next.
• Round rates and intervals to four decimal places and RICE to two.

Steps

1. In /root/founder/mvp/rates.json, write signup_rate and preorder_rate for each channel.
2. In /root/founder/mvp/mvp.py, create wilson(k, n) → [low, high] and verdict(k, n, threshold).
3. In /root/founder/mvp/verdict.json, write low, high, and verdict for each channel.
4. In /root/founder/mvp/pooled.json, write preorders, visitors, low, high, verdict for each of all (the four channels combined) and cold (the three channels combined, excluding newsletter_friends).
5. In mvp.py, add n_needed(p, threshold), and in /root/founder/mvp/sample.json, write for each channel whose verdict is insufficient n_needed (based on that channel's preorder rate) and extra_visitors (n_needed − visitors).
6. In /root/founder/mvp/rice.json, write the RICE score for each feature.
7. In /root/founder/mvp/scope.json, write included (in the order they were included), effort (the total in person-months), and excluded (in the order they were skipped).
8. In /root/founder/mvp/plan.json, write go_build (whether the cold verdict is pass), cold_verdict, next_test_channel (among insufficient channels, the one with the smallest extra_visitors), and scope_if_go (the included from step 7).

Notes

• Common mistakes: deciding on a verdict that mixes in the warm audience, comparing the rate (point estimate) directly with the threshold, multiplying confidence as 80 as is, and stopping at the first feature that exceeds the budget (you must skip it and look at the next).

Signup rate and preorder rate by channel

In /root/founder/mvp/rates.json, write signup_rate and preorder_rate for each channel.

The denominator is always the number of visitors. A channel can have a high signup rate and a low preorder rate — the hypothesis's metric is the preorder rate.

A function that judges by interval

In /root/founder/mvp/mvp.py, create wilson(k, n) and verdict(k, n, threshold).

wilson is exactly the formula from the previous module. verdict is pass if the lower bound ≥ the threshold, fail if the upper bound < the threshold, and insufficient otherwise.

The verdict per channel

In /root/founder/mvp/verdict.json, write low, high, and verdict for each channel.

The threshold is the threshold in hypothesis.json. Do not compare the point estimate directly with the threshold.

Judge again without the acquaintance channel

In /root/founder/mvp/pooled.json, write preorders, visitors, low, high, and verdict for each of all and cold.

cold is the preorders and visitors of the three channels other than newsletter_friends, added up. Add the counts; do not average the rates.

If on hold, how many more people are needed

Add n_needed(p, threshold) to mvp.py, and in /root/founder/mvp/sample.json write n_needed and extra_visitors for each insufficient channel.

Increase n from 1, recompute the interval with k = round(p·n), and stop at the first n that falls outside the threshold. The upper limit is 200000.

The RICE score

In /root/founder/mvp/rice.json, write the RICE score for each feature.

confidence_pct is a percentage, so divide by 100 before multiplying. effort is in person-months.

Scope within the budget

In /root/founder/mvp/scope.json, write included, effort, and excluded.

Go in descending score order, but skip a feature that overflows and keep looking at the next one. Break ties by name.

Build, test more, or what

In /root/founder/mvp/plan.json, write go_build, cold_verdict, next_test_channel, and scope_if_go.

Make the decision with the verdict that excludes the acquaintance channel. The next test channel is the on-hold channel that needs the fewest additional visitors.