A/B testing collection sort orders is the experiment almost no Shopify store has ever run, and it targets a bigger lever than most of the things stores do test. Merchants will split-test a headline that a fraction of visitors read, then set every collection to “Best selling” on vibes and never question it again. The sort order decides what your shoppers see first on the pages where they actually decide what to buy. It is a merchandising decision worth thousands of sessions a month, made once, tested never.
Here is why testing it matters more than your confidence does. When we analyzed the sorting recipes running across our own platform, hand-tuned sort orders beat the store’s default in only 48 percent of stores. Read that again: merchant intuition about product order is a coin flip. Half the carefully crafted sorts were worse than what they replaced, and nobody knew, because nobody measured. The full numbers are in our sorting study, and they are exactly why this tip exists: sort orders are hypotheses, and hypotheses get tested.
Sort order as an experiment
Same collection, visitors split at random, two orderings of the same products. The metric decides, not the meeting.
Variant A
Current sort
Judge on RPV per session
Variant B
Challenger sort
Same products, different shelf
One variable, chosen metric, full weekly cycles. Every rule from tips 7 and 27 applies; only the subject is new.
How to run it properly
- Pick one high-traffic collection. Your biggest collection reaches significance fastest, and its ordering is worth the most.
- Test a real hypothesis, not chaos. “Newest first will beat best-selling for our returning-heavy traffic” is a test. Shuffling products to see what happens is not.
- Judge on revenue per session for visitors who saw the collection, not clicks. A sort can win clicks with cheap products and lose the month.
- Run full weekly cycles and do not peek. Weekend shoppers are different people (tip 27). Sort tests are unusually seasonal, so what wins in October earns a re-check before Cyber Monday.
The honest obstacle: Shopify has no native way to serve two sort orders to split traffic, which is precisely why nobody runs this test. Generic A/B tools test page elements, not product order. This gap is why we built sort testing directly into Dynasort: two sorting recipes on the same collection, traffic split automatically, results scored on revenue. However you run it, run it. A coin-flip success rate on intuition is not an insult, it is an invitation: half of you are one test away from finding out your “obvious” sort order was costing you money.
