Testing in fast-moving consumer goods has its own traps. Orders are small, margins are thin, customers buy often and a lot of traffic comes from people who already know the brand. Methods copied from a fashion or electronics store can produce results that look right and aren’t. If you want ideas to test once the measurement is sound, start with these A/B test ideas for FMCG and grocery brands.
This post lists the mistakes that cause the most damage in FMCG and grocery stores, why each one distorts the result and how to avoid it.
1. Judging a test by conversion rate alone
Conversion rate is the default metric, and for FMCG it’s often the wrong one. When each order is a few dollars, the profit depends on how much is in the basket and what it costs to ship.
Imagine a test where variant B raises conversion from 4.0% to 4.2%, but average order value falls from $32 to $28. Revenue per visitor goes from $1.28 to $1.18. B “won” on conversion and lost on revenue.
What to do: Use revenue per visitor or margin per visitor as the main metric. Track conversion rate and average order value as supporting numbers, so you can see which one moved.
2. Ignoring margin and shipping cost
A discount, a lower free shipping threshold or a free gift can raise orders and still reduce profit. If shipping costs $5 per order and the product margin on a $20 order is $4, adding free shipping can turn a profitable order into a loss.
What to do: Include margin per order as a guardrail, calculated with product cost, shipping, payment fees and discounts. If you can’t calculate it precisely, estimate it per product category. A test that wins on revenue and loses on margin hasn’t won.
3. Treating repeat purchase as a quick metric
The main value in FMCG comes from customers who come back. But a repeat order may happen weeks after the first. A test that runs for two weeks can’t see it.
A subscription test shows this clearly. A default that increases signups may also bring people who cancel after the first delivery. If you stop at signups, you’ve measured only the first half.
What to do: For tests aimed at repeat behavior, decide the follow-up window in advance (for example, orders within 60 or 90 days) and keep tracking customers after the test ends. Make the decision on retention and revenue per customer, not first-order numbers. Say clearly in the test plan that the final result will arrive later.
4. Not using a stable assignment
FMCG shoppers visit often, sometimes several times a week, and from more than one device. If assignment is per session or per page view, a returning shopper can see variant A on Monday and variant B on Thursday. Both groups end up mixed, and the difference between them shrinks toward zero.
What to do: Assign at the visitor level using a persistent first-party identifier, and for logged-in customers, use the customer ID so assignment holds across devices. Document the limitation for shoppers who clear cookies or use several devices without logging in.
5. Mixing new and returning customers
In FMCG, a large part of traffic is returning customers who already know the product. They convert at far higher rates than first-time visitors, and they react differently to changes. A new shipping message might help first-timers and do nothing for loyal customers.
If your traffic mix shifts during the test (a big email send, for example), the blended result changes even if the variants have no effect.
What to do: Record the new vs. returning split in each group and check it’s balanced. Report the primary result for all visitors, then look at new and returning separately as a follow-up question. Don’t pick the segment that looks best after the fact.
6. Starting without a sample size
How much traffic you need depends on your baseline rate and the smallest effect you care about. At a 5% false positive rate and 80% power:
| Baseline conversion | Detect 10% relative lift | Detect 20% relative lift |
| 2% | about 80,700 per variant | about 21,100 per variant |
| 3% | about 53,200 | about 13,900 |
| 5% | about 31,200 | about 8,200 |
Many FMCG stores have lots of SKUs, but each product page has little traffic. A test limited to one product may never reach these numbers.
What to do: Calculate the sample size before launch, using realistic baselines such as those in these FMCG conversion rate benchmarks. New to the method? Read this guide to how A/B testing works. If it’s out of reach, test on templates or whole categories, test a larger change or measure an earlier step such as add to cart. Don’t run the test and hope.
7. Stopping when it looks good
If you check results daily and stop as soon as the difference is significant, you’ve given random noise many chances to look like a result. The real false positive rate is much higher than the 5% you intended.
What to do: Fix the sample size and end date before launch (see how long to run an A/B test). Don’t decide before both are reached. If you need to monitor as you go, use a method designed for it, such as sequential testing, rather than repeated looks at a fixed-sample test.
Run Cleaner Tests on Your Store
Set up experiments with stable assignment and revenue-per-visitor tracking without writing code. Use every CustomFit.ai feature free for 14 days.
8. Letting promotions and paydays decide the result
FMCG demand follows the calendar: paydays, weekends, holidays, weather and promotions. If one variant ran during a payday week and the other didn’t, the comparison is meaningless. Even when both run at the same time, a promotion in the middle of the test changes who’s visiting.
What to do: Always run variants at the same time. Cover whole weeks, and at least two, so each weekday appears at least twice. Note promotions, email sends and price changes on a shared calendar. If a major promotion overlaps the test, analyze it separately or extend the test.
Read more: A/B Testing for FMCG Brands on Shopify
9. Not checking for sample ratio mismatch
If you set a 50/50 split and see 10,000 and 10,400 visitors, that looks close. A chi-square test on those counts gives a p-value of about 0.005, which means the imbalance is very unlikely to be chance. Something in the setup is wrong.
Common causes include a slow variant that loses visitors before they’re counted, bot traffic, redirects that drop users and caching that serves one version more often.
What to do: Run a sample ratio check on every test, and don’t trust the results if it fails. Find the cause, fix it and restart.
10. Treating average order value as a clean number
Order values in FMCG have a long tail. Most orders are small, and a few are very large, such as a bulk stock-up before a holiday. One or two large orders can move the average of a variant noticeably.
What to do: Look at the distribution, not only the mean. Consider capping extreme orders at a high percentile (like the 99th) and state that you did. Use a bootstrap or another method suited to skewed data when computing intervals. Compare medians and the share of orders above certain values for extra context.
11. Testing too many metrics and segments
Checking ten metrics across five segments is fifty comparisons. At a 5% threshold, two or three will look significant by chance. It’s easy to pick the one that looks good and call it the finding.
What to do: Name one primary metric and a short list of guardrails before launch. Treat anything else as exploratory, to be confirmed in a follow-up test. When you must look at many comparisons, use a correction (such as Bonferroni, or control the false discovery rate), or at least raise your standard.
12. Reading a point estimate without the interval
“Variant B lifted revenue per visitor by 6%” sounds clear. With limited data, the plausible range might run from -3% to +15%. That’s a very different decision.
Also, among many tests, the ones that get declared winners tend to overstate the true effect. This is sometimes called the winner’s curse.
What to do: Report the confidence or credible interval alongside the estimate. When an interval includes effects too small to matter, say so. For important changes, re-test the winner or run a holdout after launch.
13. Ignoring the experience around the test
A testing script that delays rendering can cause a flash of the original page. A heavier variant can load more slowly. On mobile, where much grocery and FMCG shopping happens, small delays can reduce conversion by themselves.
What to do: Check each variant on real phones over a normal connection. Compare load time and Core Web Vitals between variants. If one is slower, the test is mixing design and speed.
14. Forgetting the product catalog changes
FMCG ranges change: new flavors, limited editions, discontinued lines and stock-outs. A variant that shows a popular product that sells out halfway through the test will suffer for reasons unrelated to the change.
What to do: Watch stock levels for the products involved. Note stock-outs and range changes in the test log. If a major product is unavailable for part of the test, treat the result with caution or restart.
A pre-launch checklist
- Main metric (revenue or margin per visitor) and guardrails chosen
- Sample size calculated and end date set
- Visitor-level assignment confirmed
- Variants run at the same time, for whole weeks
- New vs. returning mix recorded
- Promotions and sends on a shared calendar
- Sample ratio check planned
- Mobile rendering and speed checked
- Follow-up window set for repeat-purchase metrics
- Analysis plan written before results are seen
Summary
The most expensive FMCG testing mistakes are about measurement, not design. Judging by conversion rate when margin and basket size matter, stopping tests early, using unstable assignment and looking at repeat purchase too soon each produce results that seem solid and aren’t. Fix the measurement first, and the ideas have a fair chance to show what they’re worth. For a broader view of how this fits into conversion work, see what CRO optimization involves.
Avoid These Mistakes in Your Next Test
Begin with a sample size and one primary metric. Try CustomFit.ai free for 14 days, cancel anytime.
FAQs
Why is conversion rate misleading for FMCG tests?
Because order value and margin vary so much. A variant can raise conversion and lower basket size, leaving revenue per visitor lower. Track revenue or margin per visitor as the main metric.
How do I test something that affects repeat purchases?
Assign visitors in a stable way, set a follow-up window such as 60 or 90 days and compare reorder rates and revenue per customer. The final result arrives after the test traffic period ends.
What is a holdout group?
A group that doesn’t get the change or message, kept to show what would have happened anyway. It’s especially useful for reminders and loyalty offers, where many customers would have bought regardless.
How many visitors does an FMCG test need?
It depends on your baseline and the smallest lift you want to detect. At a 3% baseline, detecting a 10% relative lift needs roughly 53,000 visitors per variant. Calculate it for your own numbers before launch.
