Sample Ratio Mismatch (SRM): How to Detect and Fix It

Sample ratio mismatch (SRM) happens when the number of users in each variant of an A/B test differs from the split you configured by more than random chance allows. You planned 50/50, but 50,980 users landed in control and 49,020 in the variant. That gap looks small, yet it signals that something in assignment, tracking or analysis is broken, and it is one of the reasons covered in our guide to what makes a good A/B test.

SRM matters because randomization is the only thing that makes an A/B test causal. If the groups are not the size you intended, they are probably not the same kind of users either, and the measured lift can be pure bias. Large experimentation teams have reported SRM in roughly 6% to 10% of their experiments, so it is not a rare edge case.

Before you start: what you need for an SRM check

• The configured allocation: for example 50/50, or 90/10 for a cautious rollout.
• The observed counts: the number of users (not sessions or pageviews) assigned to each variant, taken from the same unit you randomized on.
• A fixed analysis window: the same start and end time for every variant.
• A significance threshold: most platforms use p < 0.001 for SRM, much stricter than the usual 0.05, because the check runs repeatedly and false alarms are costly.

1. What sample ratio mismatch means

In a correctly randomized test, each user is assigned independently with a fixed probability. The observed counts therefore follow a binomial distribution around the expected counts. Small differences, such as 10,120 vs. 9,880, are normal. SRM means the difference is too large to be explained by that randomness.

SRM is a symptom, not a cause. It tells you that the populations being compared are no longer what the randomization produced. Until you find the reason, the conversion rates, confidence intervals and p-values of the test cannot be trusted.

2. The sample ratio mismatch chi-square test

The standard SRM check is Pearson’s chi-square goodness-of-fit test. The formula is:

χ² = Σ (observed − expected)² ÷ expected

The test has (number of variants − 1) degrees of freedom. For a two-variant test, that is one degree of freedom.

Worked example 1: SRM detected

Configured split 50/50. Observed: control 50,980, variant 49,020, total 100,000. Expected: 50,000 each.

• 1. Control: (50,980 − 50,000)² ÷ 50,000 = 960,400 ÷ 50,000 = 19.21
• 2. Variant: (49,020 − 50,000)² ÷ 50,000 = 19.21
• 3. χ² = 38.42, with 1 degree of freedom
• 4. p ≈ 0.0000000006, far below 0.001

This is a clear SRM. A 51/49 split looks harmless, but with 100,000 users it is extremely unlikely to happen by chance.

Worked example 2: no SRM

Observed: control 10,120, variant 9,880, total 20,000. Expected: 10,000 each.

• 1. Each side: (120)² ÷ 10,000 = 1.44
• 2. χ² = 2.88, p ≈ 0.09

The p-value is above 0.001, so there is no evidence of SRM. The same 1.2 percentage point imbalance is normal at this sample size. That is why you should always run the test rather than eyeball the percentages.

3. SRM check for unequal splits and more variants

For a 90/10 split with 47,100 users in control and 4,900 in the variant (52,000 total), the expected counts are 46,800 and 5,200. The chi-square is (300² ÷ 46,800) + (300² ÷ 5,200) = 1.92 + 17.31 = 19.23, giving p ≈ 0.00001. That is an SRM, even though the control looks almost on target. Small arms are where SRM hides.

For A/B/C tests, add one term per variant and use two degrees of freedom. Any spreadsheet can do the calculation: in Excel or Google Sheets, CHISQ.TEST(observed range, expected range) returns the p-value directly, which turns any sheet into a sample ratio mismatch calculator.

Read more: How to Choose the Right Metrics for A/B Testing Success

4. The most common causes of SRM

SRM almost always has a technical root cause. These are the ones that show up most often in ecommerce tests:

• Redirect tests: a split-URL variant loses users when the redirect is slow or blocked, so the variant is undercounted.
• Tracking fires late or not at all: if the variant’s page loads more slowly, users bounce before the exposure event fires. Ad blockers can also affect one variant more than the other.
• Bot traffic: bots that ignore cookies or JavaScript land disproportionately in one arm.
• Changing the split mid-test: moving from 90/10 to 50/50 while keeping all users in the analysis creates SRM.
• Variant-specific bugs: a crash, a script error on one browser or a broken layout on one device type.
• Filters in the analysis: a filter that depends on behavior affected by the variant (for example “users who saw the cart”) removes users unevenly.
• Inconsistent IDs: users who switch devices or clear cookies are re-randomized and counted twice.

5. How to debug SRM step by step

• 1. Stop decision-making. Don’t declare a winner or loser while SRM is present.
• 2. Confirm the unit. Make sure you compare users with users, and that the analysis unit matches the randomization unit.
• 3. Check the timeline. Plot daily assignments per variant. A sudden break points to a deployment, a split change or a campaign.
• 4. Segment the counts. Run the SRM check by browser, device, country, traffic source and new vs. returning visitors. If one segment carries the whole imbalance, the bug is usually there.
• 5. Inspect the exposure event. Verify that it fires at the same point for every variant and before any redirect.
• 6. Look for bots. Exclude known bots and very short sessions and rerun the check.
• 7. Fix, then restart. After you fix the cause, restart the test with fresh assignment. Removing a bad segment from the analysis is acceptable only when the cause is clearly isolated and the test was expensive to run.

6. SRM in A/B testing: how to prevent it

• Assign users server-side or at the edge where possible, so tracking can’t drop users before assignment is recorded.
• Fire the exposure event at assignment, not after the page fully renders.
• Never change the allocation mid-test. To ramp up, increase exposure proportionally (for example 5/5 to 50/50).
• Run an A/A test on new tooling, which validates randomization before real experiments rely on it.
• Automate the SRM check daily and alert when p < 0.001.

Our article on common A/B testing mistakes and how to avoid them covers related errors such as peeking and stopping tests early. For background on what statistical significance does and doesn’t tell you, read this guide to statistical significance in A/B testing.

7. Running the SRM check in Python or SQL

In Python, scipy.stats.chisquare([50980, 49020], f_exp=[50000, 50000]) returns the chi-square statistic and p-value in one line. For unequal splits, pass the expected counts computed from the configured weights. In SQL, count distinct user IDs per variant from the assignment table for the analysis window, then compute the chi-square in the query or export the two counts to a sheet.

Whichever tool you use, run the check on the assignment log first and on the analysis dataset second. If the assignment log is balanced but the analysis dataset is not, the problem is in tracking, joins or filters, not in randomization. That single comparison often halves the debugging time.

8. Segment-level SRM and false alarms

Checking SRM in ten segments means ten tests, so one false alarm becomes much more likely. Treat segment checks as a diagnostic tool after the overall check fails, not as a separate alert. When one segment shows χ² far above the others, such as a single browser version, start the investigation there. When the imbalance is spread evenly across all segments, look at assignment logic or the split configuration instead.

SRM check: quick reference

Situation	Total users	Observed split	χ²	p-value	SRM?
50/50 test	20,000	50.6 / 49.4	2.88	0.09	No
50/50 test	100,000	50.3 / 49.7	3.60	0.058	No
50/50 test	100,000	51.0 / 49.0	38.42	<0.001	Yes
90/10 test	52,000	90.6 / 9.4	19.23	<0.001	Yes

SRM checklist

• Record the configured allocation before launch.
• Count users per variant using the randomization unit.
• Run a chi-square test daily with α = 0.001.
• If SRM appears, pause decisions and segment the counts.
• Fix the root cause and restart with fresh assignment.
• Never edit the split while a test is running.

If you run tests on a regular schedule, add the SRM check to the same review where you check test duration and sample size, so it becomes routine rather than an afterthought. A broader list of pitfalls is in this roundup of A/B testing mistakes that kill your results.

Where to start

Take your last three completed experiments and run the chi-square check on their user counts. It takes five minutes in a spreadsheet. If any of them show SRM, revisit the decision you made from that test, then add an automated SRM check to every future experiment.

Summary

Sample ratio mismatch means the variant sizes differ from the planned split by more than chance allows. Detect it with a chi-square goodness-of-fit test at p < 0.001, treat it as a blocker, find the root cause by segmenting the counts, and restart the test once fixed.

FAQs

Read more:

• Common A/B Testing Mistakes in FMCG Ecommerce
• Sample Ratio Mismatch (SRM) in A/B Testing