Product A/B Test: From Hypothesis to a Ship Decision
A complete experiment, not just a p-value. I designed the test, checked its validity, ran the significance testing with a guardrail, and made the call. On the public Cookie Cats dataset, moving the progression gate from level 30 to 40 significantly lowered 7-day retention, so the recommendation was clear: don't ship.
The question
Cookie Cats places a progression gate that pauses players. The experiment: does moving that gate from level 30 to level 40 change retention? Players were randomised to one version on install. The unglamorous discipline this project takes seriously is everything around the p-value: powering the test, checking the randomisation, guarding a second metric, and turning the result into a decision someone can act on.
Design, fixed before looking
Primary metric: 7-day retention. Guardrail: 1-day retention must not drop. Alpha 0.05, power 0.80. A power calculation said roughly 24,300 users per arm were needed to detect a one-point move; the data had about 45,000 per arm, so the test was well powered. Fixing all of this up front is what stops a fishing expedition after the fact.
Validity first
Before reading any result, a Sample Ratio Mismatch check: was the split close to the intended 50/50? It came out 49.6/50.4. A plain chi-square flags that as imbalanced (p=0.009), but it sits inside the established SRM alarm threshold (p<0.0005), so I treat it as a minor caveat to note, not a reason to bin the result. A failed SRM would have stopped the analysis here.
Results
Moving the gate to level 40 produced a statistically significant drop in 7-day retention. The guardrail (1-day retention) trended the same way but was not significant, an early sign rather than a clean pass.
The result, and the interval that decided it
On the left, both arms as bars on one shared axis. On the right, the difference and its 95% confidence interval against a marked zero. The interval sits entirely below zero, which is what turns a difference into a decision. Drag to orbit; click an arm to highlight it.
- Control, gate at level 30
- Treatment, gate at level 40
Primary and guardrail metrics, treatment vs control
| Metric | Control | Treatment | Difference | p |
|---|---|---|---|---|
| 7-day retention (primary) | 19.02% | 18.20% | -0.82pp | 0.0016 |
| 1-day retention (guardrail) | 44.8% | 44.2% | -0.59pp | 0.074 |
Who it hits
Broken down by engagement (exploratory, uncorrected), the drop concentrates in mid- and high-engagement players, exactly the group a game most wants to keep, while low-engagement players are unaffected.
Change in 7-day retention by engagement segment (exploratory)
| Segment | Difference | p |
|---|---|---|
| Low | +0.09pp | 0.497 |
| Mid | -0.71pp | 0.031 |
| High | -1.28pp | 0.028 |
The decision
Keep the gate at level 30; do not ship level 40. The effect is both statistically significant and practically meaningful, a real retention loss that lands hardest on engaged players, and the guardrail points the same way. A clear don't-ship recommendation, evidenced, is as valuable as a green light.
The AI-augmented workflow
I built and refactored the analysis code using an AI-augmented workflow I direct; every statistical choice was checked by hand against its assumptions before the numbers were trusted. The same rule I apply everywhere: AI to move faster, never to decide unchecked.
The pipeline was validated first on synthetic data with a known planted effect, and recovered it. That run also threw a false-positive guardrail flag where no true effect existed, kept in the writeup as a reminder that with multiple metrics you get spurious hits and should not kill a launch on one guardrail p-value without correction.