The peeking problem AB test teams run into usually starts with good intentions. Someone opens the dashboard on day three, sees “95% significant” next to the variant, and the test gets stopped so the winner can ship. Our guide on how long an A/B test should run explains why that habit is risky. This post puts hard numbers on it.
We ran a quick simulation of 20,000 tests in which the two versions were identical. Read once at the end, about 5% of them showed a false winner, which is exactly what the 95% threshold promises. Checked every day for two weeks, almost 22% did. Below we cover why that happens, which stopping rules keep you safe, and when sequential testing is the right fix for a team that has to look early.
What Is the Peeking Problem in A/B Testing?
A standard A/B test, often called a fixed-horizon test, is built to be read once. You pick a sample size, collect the data and run one significance test at the end. The 5% false positive rate on the report assumes that single look.
Peeking breaks the assumption. Every time you check the results and are ready to stop if the p-value dips under 0.05, you are running another test. Each look is a fresh chance for noise to cross the line. Over a few weeks, the running p-value of a test with no real effect will often slip below 0.05 for a day or two and then climb back up. Stop on one of those days and you ship a change that does nothing.
Looking at the data is not the issue. The issue is making a stop decision with a threshold that was never designed for repeated looks. If the p-value itself feels fuzzy, the CustomFit.ai team has a plain-English guide to statistical significance that covers it well.
Peeking A/B Testing Results: What Our Simulation Showed
To measure how big the peeking problem AB test owners face really is, we wrote a short Python simulation. The setup resembles a mid-sized store:
- Both versions convert at 3.0%, so every “winner” is a false positive.
- Each version gets 1,000 visitors a day.
- Every check uses a standard two-sided z-test with a 0.05 significance level.
- Each scenario was repeated 20,000 times.
We then compared three habits: reading the test once on the planned end date, checking once a week, and checking every day. In the peeking scenarios, the test stops at the first significant result.
| Test length | Read once at the end | Checked weekly | Checked daily |
| 7 days | 4.9% | 4.9% | 16.5% |
| 14 days | 4.7% | 8.0% | 21.7% |
| 21 days | 5.1% | 10.7% | 24.9% |
| 28 days | 4.9% | 12.3% | 27.3% |
Daily peeking on a two-week test turns a 5% error rate into roughly 22%. More than one in five tests with zero real effect would be called a winner. Weekly checks do less harm, yet by week four they still more than double the error.
Timing matters too. In the 14-day daily run, 22% of the false winners appeared on day one, and 47% appeared within the first three days. The earliest data is the noisiest, and that is exactly when people feel the strongest urge to act.
Peeking also inflates the lift you report
We ran a second version in which the variant really was better, by 5%. With daily checks over four weeks and a stop at the first significant result, 37% of the tests “won”. The median lift those winners reported was 19.7%, close to four times the true effect, and the median stop came on day 9. So even when peeking picks the right version, it oversells the result. It is the same mechanism behind the winner’s curse we described in our post on the novelty effect and fading test wins.
Read more: Minimum Detectable Effect (MDE): Formula and Examples
Early Stopping AB Test Rules That Keep Errors Near 5%
If your team reads results with a standard calculator, these six rules stop the peeking problem AB test programs keep hitting.
- 1. Fix the sample size before launch. Work it out with an A/B test sample size formula using your baseline conversion rate and MDE, then convert it into days of traffic.
- 2. Write the stop date down. Put it in the test brief next to the primary metric. A test that ends “when it looks good” has no real end date.
- 3. Run full weeks. Choose 7, 14 or 21 days, never 10. Weekday and weekend shoppers behave differently.
- 4. Look, but do not act on significance mid-test. Checking for bugs, broken tracking or a sample ratio mismatch is good practice. Stopping because p dipped under 0.05 is not.
- 5. Stop early only for harm. If a variant breaks checkout or is losing orders badly, pull it. That is a business decision, and it is the only early stopping AB test call a fixed design allows.
- 6. If you need to decide early, plan the looks in advance. This is exactly what sequential testing is for.
Rule six is where many ecommerce teams get stuck, because a festive sale or a product launch rarely waits four weeks.
What Is Sequential Testing?
Sequential testing is a way of running an A/B test that is designed for several looks. Instead of one threshold at the end, you set a stricter threshold at each planned check. The thresholds are chosen so that the overall false positive rate across all checks stays at 5%. You may stop at any planned look where the result crosses its boundary, and the 5% promise still holds.
There are two broad families. Group sequential designs are the older and simpler option, with a small number of looks fixed ahead of time, such as one per week. Always-valid methods such as mSPRT let you monitor continuously. A research team from Stanford and Optimizely explains the always-valid approach in the paper Peeking at A/B Tests: Why It Matters, and What to Do About It. The CustomFit.ai team compares SPRT, mSPRT and Bayesian options in its guide to sequential testing in A/B testing, so here we stay with the group sequential design, which you can run from a spreadsheet.
Sequential AB Testing With Weekly Looks: A Worked Example
Take the same store with 1,000 visitors per version per day. The test is planned for 28 days with a look every 7 days, four looks in total. Pocock boundaries use the same strict bar at every look. O’Brien and Fleming boundaries use a very strict bar early and a nearly normal one at the end.
| Look | Day | Naive bar, z (p) | Pocock, z (p) | O’Brien and Fleming, z (p) |
| 1 | 7 | 1.96 (0.05) | 2.36 (0.018) | 4.04 (0.00005) |
| 2 | 14 | 1.96 (0.05) | 2.36 (0.018) | 2.86 (0.004) |
| 3 | 21 | 1.96 (0.05) | 2.36 (0.018) | 2.33 (0.020) |
| 4 | 28 | 1.96 (0.05) | 2.36 (0.018) | 2.02 (0.043) |
| Overall false positive rate | 12.7% | 5.0% | 5.0% |
In our simulation, both boundary sets held the overall false positive rate at 5.0%, and they match the published values for four equally spaced looks.
What does this buy you? We gave the variant a true 15% lift. The fixed 28-day test found it 85% of the time. The O’Brien and Fleming plan found it 85% of the time as well, but it stopped by day 14 in 23% of runs and averaged 22 days instead of 28. Pocock stopped sooner, at 19 days on average, but its power fell to 78% because the final bar is stricter.
For most stores, O’Brien and Fleming is the safer default. It almost never ends a test in week one, which also protects you from the novelty spikes that make early data unreliable. You lose little if the test runs its full length.
Plan Your Stopping Rules Before Launch
Set the sample size, test length and primary metric before a single visitor sees the variant. Use every CustomFit.ai feature free for 14 days.
When Sequential Testing Is Worth It
Sequential designs earn their keep in four situations:
- Time-boxed windows. A ten-day sale leaves no room for a four-week test, and planned looks let you act inside the window.
- Costly variants. If a paid landing page test is burning ad budget, a clear loser should not run for another three weeks.
- Teams that will check daily anyway. A planned design beats pretending nobody looks.
- High test volume. Saving six days per test across 20 tests a quarter frees up months of traffic.
They help less elsewhere. Sequential testing does not create data, so a store that needs six weeks for a fixed test will rarely stop early unless the effect is large. Tests that target long-term behavior, such as loyalty changes, should run their full length regardless. And a team that does not write down its boundaries before launch is simply peeking with extra steps.
Bayesian tools are often sold as safe to peek at. They are more forgiving, but a rule such as “stop when the probability to beat control reaches 95%”, checked every day, still produces more wrong calls than that number suggests. The CustomFit.ai team covers the trade-offs in Bayesian vs frequentist A/B testing. Evan Miller’s classic essay How Not To Run an A/B Test is still the clearest short explanation of why repeated significance checks mislead.
Peeking Problem AB Test Checklist
- Write the primary metric, sample size and end date into the test brief before launch.
- Run full weeks, and never fewer than 14 days for changes that returning shoppers will notice.
- Check daily for bugs, tracking gaps and sample ratio mismatch, but not for significance.
- Stop early only for harm, and record the reason.
- If you need early decisions, choose sequential boundaries before launch and stick to them.
- Treat large lifts from tests that stopped early with caution, and expect a smaller effect after rollout.
Run Tests You Can Trust
Start with one high-traffic page, a fixed end date and a single primary metric. Try CustomFit.ai free for 14 days, no credit card required.
Summary
Peeking problem AB test errors come from statistics, not from a lack of discipline. A 5% threshold is valid for one look, and every extra look adds risk. In our simulation, daily checks on a two-week test raised false positives from 5% to about 22% and made real wins look several times larger than they were. The fix is either to wait for the planned sample size or to plan your looks up front with sequential boundaries such as O’Brien and Fleming, which keep the error at 5% while still allowing an early stop.
FAQs
What is the peeking problem in an A/B test?
The peeking problem AB test teams hit happens when you check a running test repeatedly and stop as soon as it reaches significance. Each check is another chance for random noise to cross the 0.05 line, so the real false positive rate climbs well above the 5% you think you are accepting.
How much does peeking at A/B testing results inflate false positives?
In our simulation of identical versions with 1,000 visitors per version per day, daily checks raised the false positive rate to about 17% over 7 days, 22% over 14 days and 27% over 28 days. Reading the result once at the end kept it near 5%.
What is sequential testing?
Sequential testing is a test design that allows several planned looks at the data. Each look uses a stricter significance boundary, chosen so the overall false positive rate stays at 5%. You can stop at any look where the result crosses its boundary without inflating the error.
Can you stop an A/B test early?
Yes, in two cases. You can stop for harm, such as a broken checkout or a large drop in orders. You can also stop for a win, but only if the test was planned as a sequential test with boundaries set before launch. Stopping a fixed test early because it looks significant is the classic peeking problem AB test mistake.
Is sequential A/B testing better than a fixed-horizon test?
Not always. Sequential AB testing helps when you face a deadline, a costly variant or a team that will check results often. A fixed test is simpler and works well when you can wait. With O’Brien and Fleming boundaries, you lose very little power and often finish a week earlier.



