The minimum detectable effect is the smallest change in a metric that your A/B test can reliably pick up. Set it too small and the test needs more visitors than you will ever get. Set it too large and you will miss real improvements and call them failures. Many teams pick a number in passing, yet it decides how long every test runs and what it can tell you.
This guide explains what the minimum detectable effect means for MDE A/B testing, the formula behind it, how to work it out from your own traffic, and how to choose a sensible value for an ecommerce store. It includes a lookup table and a worked example.
What is the minimum detectable effect?
When you plan a test, you make a bet about how big the difference between control and variant needs to be before you care about it. That threshold is the minimum detectable effect, usually shortened to MDE. If the true effect is at least that large, a test with the planned sample size has a good chance (normally 80%) of showing a statistically significant result. If the true effect is smaller, the test will often come back flat, even though something real happened.
It helps to think of MDE as the resolution of your test. A test with a 5% MDE is a fine lens. It can see small shifts, but it needs a lot of light, which here means traffic. A test with a 25% MDE is a coarse lens. It needs less traffic, but only big changes show up.
Relative vs absolute MDE
MDE can be written two ways, and mixing them up is one of the most common planning errors.
- Absolute MDE: the change in percentage points. Moving conversion from 3.0% to 3.6% is an absolute MDE of 0.6 points.
- Relative MDE: the change as a share of the baseline. The same move from 3.0% to 3.6% is a relative MDE of 20%.
Most calculators want relative MDE, but many dashboards show absolute differences, so check which one a tool expects. A relative MDE of 2% on a 3% baseline means detecting a move to 3.06%, which needs a huge sample. An absolute MDE of 2 points means detecting a move to 5%, which needs very little.
The four inputs that set your MDE
MDE is not a free choice. It is tied to four other numbers, and if you fix any three, the fourth follows.
- 1. Baseline conversion rate: how often visitors on the test page already do the thing you’re measuring. The lower it is, the more traffic you’ll need to spot the same relative change.
- 2. Significance level (alpha): the false positive rate you accept, usually 5%, which matches 95% confidence.
- 3. Statistical power: the chance of detecting a real effect of MDE size, usually 80%.
- 4. Sample size per variant: how many visitors each version gets before you read the result.
This is why MDE vs sample size is really one question asked from two directions. You either pick an MDE and calculate the sample you need, or you start from the traffic you have and calculate the smallest effect you can detect. Our guide to the A/B test sample size formula covers the first direction in depth. This post focuses on the second, because for most stores traffic is the hard limit.
The minimum detectable effect formula
For a two-arm test on a conversion rate, the standard sample size formula per variant is:
n = (z_alpha/2 + z_beta)^2 x [p1(1 – p1) + p2(1 – p2)] / (p2 – p1)^2
Here p1 is the baseline rate, p2 is the rate you want to detect, z_alpha/2 is 1.96 for 95% confidence and z_beta is 0.84 for 80% power. Together those two z values add up to about 2.80, and squared they give roughly 7.85.
To get the MDE, you turn the formula around. Since p2 appears on both sides, there is no neat closed form, so calculators solve it by trial. A quick approximation is good enough for planning:
Absolute MDE = 2.80 x square root of [2 x p1(1 – p1) / n]
Divide the absolute MDE by p1 to get the relative MDE. It runs slightly low for large effects but is fine for planning.
How to calculate MDE: a worked example
Say your product page converts at 3%, and about 20,000 visitors a week reach it. You plan to split traffic 50/50 and run the test for four weeks.
Four weeks of 20,000 visitors split two ways gives each arm 40,000 people. Plug the 3% baseline in and the variance part, 2 x 0.03 x 0.97, comes to 0.0582. Divided by 40,000 that’s tiny, about 0.0000015, and its square root lands near 0.0012. Times 2.80, you get roughly 0.0034. So the test can see a jump of about a third of a percentage point, which on a 3% base works out to around 11% relative.
Solved exactly, it comes out at 11.6%. So this test can reliably detect a lift from 3.00% to about 3.35% or larger. If you expect your change to move conversion by 3% or 4%, this test is not built to see it.
Read more: A/B Test Hypothesis Template (with 12 Ecommerce Examples)
MDE lookup table for ecommerce tests
The table shows the relative MDE at 95% confidence and 80% power for common baselines and sample sizes. Find your baseline, then the number of visitors each variant will get during the test.
| Baseline conversion | 10,000 per variant | 25,000 per variant | 50,000 per variant | 100,000 per variant |
| 1% | 43.4% | 26.5% | 18.4% | 12.9% |
| 2% | 29.7% | 18.3% | 12.8% | 9.0% |
| 3% | 23.8% | 14.7% | 10.3% | 7.2% |
| 5% | 18.0% | 11.2% | 7.9% | 5.5% |
Two patterns stand out. Doubling the traffic does not halve the MDE. It shrinks it by about 30%, because the MDE falls with the square root of the sample. Also, low-converting pages are expensive to test. A checkout step that converts at 50% is easy to measure; a category page at 1% add-to-cart is not.
Any minimum detectable effect calculator uses these same inputs. The CustomFit.ai A/B testing sample size calculator shows the visitors you need for a given MDE.
Plan Tests Your Traffic Can Actually Detect
Set up product page and checkout experiments without code, then let them run to a planned sample size. Use every CustomFit.ai feature free for 14 days.
How to choose an MDE that makes sense
There is no universal right value. Pick an effect worth having that you can detect in reasonable time.
Start from the business, not the calculator
Ask what lift would justify the work of building and maintaining the change. If a new size guide takes two weeks of design and development, a 1% lift might not repay it. A 10% lift probably would. That value is your business MDE. If a test cannot detect it, either the test needs more traffic or the change needs to be bolder.
Check it against your traffic
Next, use your weekly visitors and a maximum test length to find the smallest MDE you can afford. Most teams cap tests at four to six weeks, since longer runs pick up cookie churn, seasonal swings and overlapping promotions. Our guide on how long to run an A/B test covers these limits in more detail.
Match the MDE to the size of the change
Small copy tweaks and button color tests tend to move metrics by a few percent at most. Structural changes, such as a new product page layout, a different offer or a rebuilt checkout, can move them by 10% to 30%. On a store with modest traffic, the honest answer is often to test bigger ideas, because small ones fall below the MDE you can reach.
Pick the right metric
The MDE also depends on what you measure. A metric close to the change, such as add-to-cart rate on the product page, usually has a higher baseline and a larger effect than final orders, so it reaches a usable MDE faster. Revenue per visitor is noisier and normally needs a bigger sample. Choosing metrics carefully, as covered in our post on choosing the right metrics for A/B testing, often does more for MDE than any amount of extra traffic.
Common mistakes with MDE in A/B testing
- Choosing an MDE after the test starts. If you shrink the MDE once you see a small lift, you are fitting the plan to the result. Fix it in your test brief before launch.
- Reading a flat result as no effect. A non-significant test only tells you the effect was probably smaller than your MDE. It does not prove the change did nothing.
- Forgetting about multiple variants. Every extra arm splits the traffic further. Three variants plus control means each arm gets a quarter of visitors, so the MDE rises noticeably.
- Raising power without adjusting the timeline. Moving from 80% to 90% power increases the required sample by about 34% for the same MDE.
- Trusting a winner far bigger than the MDE. When a low-traffic test shows a 40% lift, the true effect is often much smaller. Significant results on underpowered tests tend to exaggerate the effect size.
The guide to statistical significance in A/B testing explains why these results look better than they are and how confidence intervals help you read them more fairly.
A quick MDE checklist for your next test
- Write down the baseline rate for the primary metric on the exact page and audience in the test.
- Decide on 95% confidence and 80% power unless you have a clear reason to change them.
- Set a business MDE: the smallest lift worth shipping.
- Calculate the traffic MDE from weekly visitors and a four to six week cap.
- If the traffic MDE is larger than the business MDE, make the change bolder, test a higher-traffic page or pick a closer metric.
- Record the MDE, sample size and stop date in the test plan before launch.
Turn Your MDE Into a Test Plan
Choose the page, set the split and let each test run to the sample size it needs. Try CustomFit.ai free for 14 days, no credit card required.
Summary
The minimum detectable effect is the smallest real change a test can reliably find, and it is set by your baseline rate, confidence level, power and sample size. Work it out before every test, using the formula or a calculator, and compare it with the lift that would make the change worthwhile. When the numbers do not line up, test bolder ideas or better metrics instead of running longer and hoping.
FAQs
What is a good minimum detectable effect for an A/B test?
For most ecommerce tests a relative MDE between 5% and 20% is realistic. High-traffic stores can aim for 5% or lower, while smaller stores usually need 15% to 25% to finish a test within four to six weeks. Choose a value that is both worth shipping and reachable with your traffic.
How do you calculate the minimum detectable effect?
Take your baseline rate, the confidence and power you want, and the visitors each variant will get, then work the sample size formula backwards. For a rough number at 95% confidence and 80% power, multiply 2.80 by the square root of 2 x p(1 – p) / n. That gives you the change in percentage points.
How is MDE different from statistical significance?
You set the MDE before launch. It’s the smallest lift your test is built to catch. Significance comes at the end, once the data is in, and tells you whether the gap you saw is bigger than random noise would usually produce. One is a design choice, the other is a verdict.
Does a lower MDE need a bigger sample size?
Yes. Required sample size grows with the inverse square of the MDE, so halving the MDE roughly quadruples the visitors you need. That is why tests aimed at very small lifts often run for months on stores with moderate traffic.
Should I use relative or absolute MDE?
Use whichever your calculator expects, but always say which one you mean. Relative MDE is easier to compare across pages with different baselines, while absolute MDE in percentage points is clearer when you report the expected change in conversion rate.
