Inconclusive AB Test? 6 Things to Do Next

Inconclusive AB test illustration with confidence intervals crossing zero and a magnifying glass with a question mark on a glass panel above a laptop showing A and B test panels.

An inconclusive AB test is the most common result in ecommerce experimentation, and it is also the one teams handle worst. The test ran for three weeks, the dashboard says “not significant”, and someone in the meeting asks whether the variant “kind of won”. It did not, but it did not lose either. What you do next depends on a question most reports never answer: did the change have no effect, or was the test too small to see one?

This guide shows how to tell those two cases apart with the numbers you already have, and then gives six practical moves for the next test. If you are still planning a test, our explainer on minimum detectable effect for ecommerce tests will save you from many flat results before they happen.

Why an Inconclusive AB Test Happens

Most flat tests come from one of four causes. Knowing which one you are dealing with is half the diagnosis.

  • The change was too small for shoppers to notice. A new shade of button, a reworded subheading or a badge moved 40 pixels rarely shifts buying behaviour enough to measure.
  • The test was underpowered. The sample was only big enough to detect a large lift, so a real but modest effect stayed hidden in the noise.
  • The metric was too far from the change. You edited the product description but judged the test on completed orders, which sit three steps away and depend on shipping costs, payment options and stock.
  • The effect really is close to zero. Some ideas simply do not matter to buyers. That is a valid finding, not a failure.

Large testing programs regularly report that only a minority of experiments produce a clear win. A run of flat results is normal, not a sign that something is broken.

First, Rule Out a Broken Test Before Calling an A/B Test Inconclusive

A surprising share of the times we see an A/B test inconclusive, the test itself was faulty. Before interpreting anything, we run through this list:

  • 1. Did the test reach its planned sample size? If you stopped early because the line looked flat, you have not finished the test. Our post on the peeking problem in A/B testing explains why early looks mislead in both directions.
  • 2. Is the traffic split balanced? A 50/50 test that ended 52/48 on 80,000 visitors points to a tracking or redirect bug, and the result cannot be trusted.
  • 3. Did both variants track conversions the same way? Fire a test order through each variant and confirm the event lands in your analytics.
  • 4. Did the test cover at least one full weekly cycle, ideally two?
  • 5. Did anything unusual happen during the test? A flash sale, a stockout on a best seller or a viral social post can swamp a small effect.

If any check fails, fix the problem and rerun.

How to Read a “No Significant Difference” AB Test Result

A p-value above 0.05 tells you one thing: the data does not rule out zero. It does not tell you the effect is zero. To learn which situation you are in, look at the confidence interval for the lift, not just the verdict.

Take two stores, both with a 3.0% baseline conversion rate.

Store A sent 40,000 visitors to each arm. The variant converted at 3.09%, a 3% relative lift, with a p-value of 0.46. The 95% confidence interval for the relative lift runs from about minus 5% to plus 11%.

Store B sent 400,000 visitors to each arm. The variant converted at 3.02%, a 0.7% relative lift, with a p-value of 0.60. Here the interval runs from about minus 2% to plus 3%.

StoreVisitors per armObserved lift95% interval for the liftWhat it means
A40,000+3%about -5% to +11%Underpowered: a 10% win is still possible
B400,000+0.7%about -2% to +3%Likely no meaningful effect

Both tests report no significant difference, yet they say very different things. Store A cannot rule out a 10% lift, which would be worth a lot. Store B can rule out anything larger than about 3%, so if the team only cares about lifts of 5% or more, the idea is effectively dead.

Flat AB Test Results: No Effect or Underpowered?

Most flat AB test results fall into one of three buckets, and a simple rule sorts them. Before the test, write down the smallest lift worth shipping, the same number you used for your MDE. After the test, compare that number with the confidence interval.

  • The whole interval sits below your smallest worthwhile lift. Treat it as no meaningful effect. Stop investing in this idea.
  • The interval includes your smallest worthwhile lift. The test was underpowered for the question you asked. You have learned less than you think.
  • The interval is mostly negative. The variant probably hurts. Keep the control and record why.

You can also check power after the fact by looking back at your plan. Store A’s test, at 40,000 visitors per arm, had roughly 68% power to detect a 10% lift and only about 11% power to detect a 3% lift. Detecting a 3% lift at 80% power would have needed about 570,000 visitors per arm. A quick run through our A/B test sample size formula or the Evan Miller sample size calculator shows the gap in seconds.

If you need a formal answer to “is the effect small enough to ignore?”, the statistical tool is an equivalence test, which flips the usual question and tests whether the lift sits inside a range you define as negligible.

Read more: A/B Test Hypothesis Template (with 12 Ecommerce Examples)

Plan Tests That Can Actually Detect a Lift

Set your baseline, smallest worthwhile lift and traffic before launch, then run the test without code. Try CustomFit.ai free for 14 days.

Try CustomFit.ai Free No credit card required

What to Do With Inconclusive Test Results: 6 Next Moves

Once you know whether you are looking at “no effect” or “not enough data”, pick one of these moves. They are listed roughly in the order we try them.

1. Test a Bigger, Bolder Change

Small tweaks produce small effects, and small effects need enormous samples. If the interval was wide, the cheapest fix is often a bigger swing. Instead of rewording one line of the product description, test a fully rewritten page section built around the main objection buyers raise in reviews. Instead of moving a trust badge, test a full returns guarantee block beside the price. A change that could plausibly move conversion by 15% needs roughly a tenth of the traffic that a 5% change needs.

2. Move the Test to a Higher-Traffic Page

If the idea is sound but the page is quiet, run it where the traffic is. A sizing prompt tested on one product page with 3,000 weekly visitors may never finish. The same prompt applied as a template change across all apparel product pages might see 60,000 visitors a week and reach a verdict in under a month.

3. Choose a Metric Closer to the Change

A product page change should usually be judged on add to cart rate first, with orders and revenue per visitor as guardrails. Metrics close to the change have higher base rates and less noise from later steps. With a 9% add to cart rate and 40,000 visitors per arm, the smallest lift you can reliably detect drops to about 6%, compared with about 11% for a 3% purchase rate on the same traffic. The CustomFit.ai guide to primary, secondary and guardrail A/B testing metrics lays out how to set this hierarchy before launch.

4. Check Pre-Planned Segments, Carefully

Sometimes a flat overall result hides a mobile gain and a desktop loss. Segment analysis can reveal that, but it is also the fastest way to fool yourself. Slice the data ten ways and one slice will look significant by chance. Only trust segments you named before the test, and treat any segment “win” as a new hypothesis to confirm in a fresh test, not as a reason to ship.

5. Decide on Cost When the Effect Is Negligible

If the interval shows the effect is close to zero, the decision is no longer statistical. Pick the version that is cheaper to run. A variant that removes an expensive third-party review widget without hurting conversion is a real win on margin and page speed. A variant that adds design debt for no gain should be dropped.

6. Log It as a Learning and Move On

An inconclusive result still narrows the search. “Shoppers in this category do not respond to urgency messaging at the levels we tested” is useful knowledge that should shape the backlog. Record the hypothesis, sample size, interval and what you believe it means. The CustomFit.ai team covers this follow-up process well in its post on how to recover from a failed A/B test, including how to turn a flat result into a stronger next hypothesis.

What Not to Do After a Flat Test

A few habits turn inconclusive results into bad decisions:

  • Extending the test without a plan. Adding “one more week” until the number turns green is peeking under another name. Extend only if you never reached the planned sample.
  • Shipping the variant because it was slightly ahead. A 3% lift with an interval from minus 5% to plus 11% is a coin flip, not a win.
  • Calling the idea dead after an underpowered test. Store A cannot conclude anything about a 3% effect. Saying “we tested that, it didn’t work” closes a door that was never opened.
  • Switching the primary metric after the fact. Picking whichever metric moved most inflates false positives.

Turn Your Next Flat Test Into a Clear Answer

Begin with a bolder variant or a higher-traffic template. Try CustomFit.ai free for 14 days, no credit card required.

Try CustomFit.ai Free No credit card required

Summary

An inconclusive AB test means the data could not rule out zero. It does not mean the change did nothing. Start by checking the test was sound: planned sample reached, balanced split, working tracking and full weekly cycles. Then compare the confidence interval with the smallest lift worth shipping. A narrow interval near zero means no meaningful effect, so decide on cost and move on. A wide interval means the test was underpowered, so try a bigger change, a busier page or a closer metric. Either way, log what you learned so the next test starts smarter.

FAQs

What does an inconclusive AB test mean?

Short version: you ran the test and got no verdict. The gap you saw between A and B was small enough that it could easily be zero, so you can’t call either version the winner yet. It does not prove the change had no effect. Check the confidence interval: if it is wide, the test was likely underpowered; if it is narrow and centred near zero, the effect is probably negligible.

Should I run an A/B test longer if it is inconclusive?

Only if the test has not yet reached the sample size you planned before launch. Extending a finished test until it turns significant is a form of peeking and raises your false positive rate. If you already hit the planned sample, design a new test with a bigger change, more traffic or a closer metric instead.

How do I know if flat AB test results mean no effect?

Compare the confidence interval for the lift with the smallest lift worth shipping. If the whole interval sits below that threshold, treat the result as no meaningful effect. If the interval still includes it, the test was underpowered and cannot tell you whether the idea works.

Is no significant difference in an AB test the same as no difference?

No. A non-significant result only means the observed difference could plausibly be due to chance at your sample size. A small test can miss a real 10% lift. To show that two variants perform about the same, you need a narrow confidence interval or a formal equivalence test.

What to do with inconclusive test results that look promising in one segment?

Treat the segment result as a new hypothesis, not a winner. Unless the segment was named before launch, it may be a chance finding from slicing the data many ways. Run a fresh test targeted at that segment with its own sample size calculation, and ship only if the effect holds.

Read more:

Leave a Comment

Your email address will not be published. Required fields are marked *