To A/B test a collection sort order, run two sorting approaches against each other on the same collection, split exposure between them fairly, hold everything else constant, and judge the winner on collection conversion rate. Shopify has no built-in way to test sort orders, so in practice you either alternate sorts on a schedule or use an app that automates the assignment and the measurement.
What Should You Hold Constant?
A sort order test is only valid if the sort order is the only thing changing. Keep these fixed for the full duration:
- Collection membership. Do not add or remove products mid-test.
- Prices and promotions. A discount that runs during only one variant’s exposure decides the test for you.
- Theme and layout. No grid, filter, or template changes while the test runs.
- Pins. If certain products are pinned to fixed positions, pin them identically in both variants.
How Do You Split Exposure?
The two practical methods are visitor splits and time-based alternation. Alternating the sort on a short cycle, for example switching every hour, is often the simpler and surprisingly robust option: both variants see the same days of the week, the same campaigns, and the same seasonality, which kills the biggest sources of bias in before-and-after comparisons.
Measure collection conversion rate per variant as the primary metric, and watch click behavior as a secondary signal: a better sort usually pulls clicks toward the first rows because shoppers find what they want sooner. Decide in advance how long the test will run, then let it finish. In Dynasort, recipe A/B testing is included on the Pro and Enterprise plans, with the rotation and per-variant measurement handled for you.
Dynasort runs sort order tests like this automatically. Install it from the Shopify App Store or see how it works.