A collection A/B test should run until each variant has accumulated enough conversions to separate a real difference from random noise, and for at least one or two full weekly cycles so weekday and weekend shoppers are both represented. For most collections that means weeks, not days. Set the end condition before you start, and do not stop early because one variant pulls ahead.
Why Conversions, Not Days, Decide
The unit that determines how trustworthy a test is, is conversions per variant, not elapsed time. A collection that converts hundreds of orders a week can reach a readable result quickly; a collection with a handful of weekly orders may need a month or more to show anything beyond coin-flip variance. That has two practical consequences:
- Test your higher-traffic collections first. They answer faster, and what you learn usually transfers to smaller collections.
- Small differences need long tests. If two sort orders perform within a whisker of each other, you may never get a clean winner, and that is itself an answer: pick the simpler option and move on.
What Cuts a Test’s Validity?
Most invalid tests die from impatience or interference rather than math. The common failures: stopping the moment a variant looks like it is winning (early leads flip constantly), changing prices or collection membership mid-test, running across an unrepresentative event like a flash sale or BFCM, and comparing periods with different traffic mixes, such as a paid-campaign week against an organic week. Time-based alternation, where the two sorts swap on a short cycle, protects against the traffic-mix problem because both variants experience the same conditions.
When the test ends, act on the result, then keep measuring. A winning sort order is a snapshot, and catalogs change. If you want the rotation and the math handled for you, recipe A/B testing is part of Dynasort’s Pro and Enterprise plans.
Dynasort handles the test from setup to readout. Install it from the Shopify App Store or see how it works.