Knowing when NOT to A/B test is as much a CRO skill as knowing how to run one. This series has spent several tips teaching testing discipline, so here is the counterweight: testing has a cost. Every test takes calendar time, splits your traffic, and occupies the one experiment slot most small stores can honestly run. Spending that on a question that does not deserve it is not rigor. It is procrastination with a dashboard.
Three categories never deserve a test.
Should this be a test?
Run every idea through four gates, in order. Most never reach the last one.
Gate 4 is rare. That is the point: tests are for genuine uncertainty, not for outsourcing every decision.
The traffic gate is the one merchants skip
Gates 1 and 2 are common sense once said aloud. Gate 3 is where stores lie to themselves. A meaningful A/B test needs enough conversions, not sessions, conversions, in each variant to separate signal from noise, and a store doing a handful of orders a day simply cannot resolve a subtle change in a few weeks. Run the test anyway and you get a coin flip wearing a lab coat: whichever variant got lucky “wins”, you roll it out with confidence you have not earned, and the next debate cites it as evidence.
If that is your traffic reality, you are not excused from rigor, you just use a different kind: ship the change on a known date, annotate it, and compare a few weeks before against a few weeks after on the baseline you have kept since tip 1. It is weaker evidence than a controlled test and dramatically stronger than vibes.
The hot take, 24 days out: right now, bias toward shipping. A testing culture is a luxury of February. In November, an obvious improvement sitting in a queue behind a three-week experiment is revenue you volunteered to lose. Save the lab for questions that deserve it; the freeze on new experiments is coming in tip 80 anyway.
