Losing A/B tests teach more than winning ones, and almost every store throws the lesson in the trash. A winner gets shipped, screenshotted, and mentioned in three meetings. A loser gets quietly rolled back and never spoken of again, which is a shame, because the loser is the one carrying new information: it just told you that something you believed about your customers is false.

A winning test mostly confirms what you already suspected; that is why you ran the variant. A losing test, especially the “obvious” winner that lost, is your customers correcting your model of them. That correction compounds: it does not just kill one variant, it should redirect every future test you were about to build on the same wrong belief. Expert intuition loses more often than experts think. When we compared hand-tuned collection sorting against boring defaults across our own fleet, the hand-tuned versions won in only 48% of stores. A coin flip, from people who know their catalogs cold. Your homepage hunches are not doing better.

Why the log matters more than the memory

Unlogged losers do not stay dead. Teams re-run failed ideas 18 months later because nobody remembers the funeral, and memory launders every result: “it lost” becomes “the timing was off” becomes “we should try it again for BFCM.” The log is the antidote. It also protects your winners from over-generalization: “free shipping messaging won on the cart page in October” is a fact; “urgency works on our customers” is a myth in the making, and the log keeps the fact-sized version on record.

One habit upgrade while you are at it: write the “belief updated” line for losers with the same energy you would give a win. The test that cost you two weeks and proved your best idea wrong did not fail. It bought you the one thing traffic cannot: a truer model of the people you sell to. That is tuition, and you already paid it. Keep the receipt.

A/B tests started after mid-October are running on borrowed truth. This is last call: any big test you want to trust through Q4 needs to start this week, because the traffic that will flow through it from here on is progressively less like the traffic your store lives on the rest of the year.

The contamination is not one event, it is a slow tide. From late October your mix shifts: more gift buyers shopping for someone else, more first-time visitors from holiday ads, more deal-hunters comparison shopping across ten tabs, more mobile sessions squeezed into commutes and couches. These people click differently, carry different intent, and respond to different persuasion than your September regulars. A variant that wins with them has proven it wins with them, in a mix that exists six weeks a year.

Triage your test backlog today

• Start now: anything big, structural, and reversible. Layout changes, PDP restructures, the sort-order test from tip 49. Two full weekly cycles still fit before the tide comes in.
• Park until January: anything you cannot start this week. An honest backlog beats a contaminated result, and January traffic is the closest thing to a control group you get all winter.
• Exempt from all of this: obvious fixes and broken things. Those are not tests, just ship them (that argument gets its own tip later in the series).

The deeper point: a test is a measurement instrument, and instruments are only valid in the conditions they were calibrated for. From here to Cyber Monday your store’s conditions change weekly, which is why the series keeps splitting testing into two seasons: the lab season, which is ending, and the playbook season, which is about to begin. Tip 80 marks the full freeze. This week is the last comfortable boarding call before it.

A/B testing collection sort orders is the experiment almost no Shopify store has ever run, and it targets a bigger lever than most of the things stores do test. Merchants will split-test a headline that a fraction of visitors read, then set every collection to “Best selling” on vibes and never question it again. The sort order decides what your shoppers see first on the pages where they actually decide what to buy. It is a merchandising decision worth thousands of sessions a month, made once, tested never.

Here is why testing it matters more than your confidence does. When we analyzed the sorting recipes running across our own platform, hand-tuned sort orders beat the store’s default in only 48 percent of stores. Read that again: merchant intuition about product order is a coin flip. Half the carefully crafted sorts were worse than what they replaced, and nobody knew, because nobody measured. The full numbers are in our sorting study, and they are exactly why this tip exists: sort orders are hypotheses, and hypotheses get tested.

How to run it properly

• Pick one high-traffic collection. Your biggest collection reaches significance fastest, and its ordering is worth the most.
• Test a real hypothesis, not chaos. “Newest first will beat best-selling for our returning-heavy traffic” is a test. Shuffling products to see what happens is not.
• Judge on revenue per session for visitors who saw the collection, not clicks. A sort can win clicks with cheap products and lose the month.
• Run full weekly cycles and do not peek. Weekend shoppers are different people (tip 27). Sort tests are unusually seasonal, so what wins in October earns a re-check before Cyber Monday.

The honest obstacle: Shopify has no native way to serve two sort orders to split traffic, which is precisely why nobody runs this test. Generic A/B tools test page elements, not product order. This gap is why we built sort testing directly into Dynasort: two sorting recipes on the same collection, traffic split automatically, results scored on revenue. However you run it, run it. A coin-flip success rate on intuition is not an insult, it is an invitation: half of you are one test away from finding out your “obvious” sort order was costing you money.

A/B test sample size is the part of testing nobody wants to talk about, because the math says most small stores cannot run most of the tests they are running. There, we said it. A test’s job is to detect a difference, and detection has a price paid in visitors. Small lift, small traffic: the test cannot pay, and it will hand you a coin flip dressed up as a result.

The intuition is simple. Conversion data is noisy; any two identical pages will show different numbers over a week just by chance. To claim your variant “won,” its lift has to stand out above that noise, and the noise only shrinks with volume. The smaller the true lift, the more visitors you need before you can tell it apart from luck, and the relationship is brutally nonlinear: halving the lift you want to detect roughly quadruples the traffic you need. A subtle button tweak that might move conversion a few percent relative is, on a store doing a hundred orders a month, statistically invisible. You could run it for a quarter and learn nothing except that winter happened.

Run the math before the test

Before you build anything, spend two minutes in a sample size calculator: enter your baseline conversion rate and the smallest lift you would care about, and it tells you the visitors needed per variant. Divide by your weekly traffic to the tested page. If the answer is “nineteen weeks,” you have your verdict: that test is not runnable on your store. That is not a failure, it is a free answer that just saved you a month.

What small stores should do instead

• Test bigger swings. Big changes produce detectable effects on small traffic. New offer structure, different hero, restructured product page: these can clear the noise floor. Button colors cannot.
• Test upstream, where the events are. This is why we told you in tip 33 to watch add-to-cart rate: more events per visitor means the same traffic buys you more statistical power than purchase-based tests get.
• Test on your busiest pages. The homepage and your top collection can support tests your long-tail product pages never will.
• Below the traffic floor, stop pretending. Make the change, watch your baselined metrics for two full weeks, and be honest that this is a before/after judgment, not an experiment. A humble before/after beats a fake A/B test, because at least you know how much to trust it.

The discipline from tip 7 still stands: one variable, a metric chosen in advance, no peeking. Today’s rule sits in front of it: first check whether the test can be won at all. A 5 percent lift you cannot detect is a test you should refuse to run.

A/B tests should run in full weekly cycles, ending on the same weekday they started, and this one scheduling rule quietly invalidates more merchant experiments than any statistics mistake. Your store does not have one audience. It has a Tuesday-lunch-break audience and a Sunday-evening-couch audience, and they are different people buying in different moods with different wallets.

Look at your own analytics by day of week and you will see the fingerprint: weekday traffic that browses and bounces, weekend traffic that settles in and converts, or the exact reverse if you sell office supplies. Payday weeks spike. Sunday evenings buy differently than Friday nights. None of this is noise. It is a weekly rhythm as real as your opening hours used to be.

Now run a test from Wednesday to Monday. Variant A got four weekdays and one weekend; anything that appeals to your weekend buyer is undercounted, anything that suits the weekday skimmer is overcounted. You did not measure “A vs B”, you measured “A on these days vs B on those days” and called it science.

The rules, in full

• Run whole weeks. Seven days, or fourteen, or twenty-one. Start Tuesday, end Tuesday. Every weekday appears the same number of times in both variants.
• Never end on a spike. The urge to call the test the morning after a great day is exactly backwards: a spike is the moment your sample is least representative. If the result is real, it will still be there when the cycle completes.
• Never end on a dip either. Killing a variant during a slow Monday is the same coin flip with sad music.
• Decide duration before you start, like you decided the success metric in tip 7. A stopping rule chosen mid-test is not a rule, it is a mood.
• Quarantine abnormal weeks. A payday-sale week, a press mention, a shipping meltdown: if the week was weird, extend the test by a full cycle rather than reading tea leaves.

This matters double as Cyber Monday approaches, because Q4 compresses everything: traffic climbs week over week, and the temptation to run quick three-day tests grows just as the weekly rhythm gets loudest. Resist. A clean two-week answer in September beats a corrupted three-day answer in November, and September is exactly when you still have time to act on what you learn.

Patience is not the glamorous part of optimization. It is the part that makes the rest of it true.

Prioritize your A/B tests by traffic times impact times ease, not by whoever argues loudest in the meeting. Every store has more test ideas than traffic to run them, which makes prioritization the actual skill. The store that runs its best idea first beats the store that runs its founder’s favorite, every quarter, forever.

The scoring takes ten minutes. For each idea, rate three things from 1 to 5: Traffic, how many sessions actually see the thing you are changing; Impact, how big a behavior change you honestly expect if it wins; Ease, how cheap it is to build and undo. Multiply. Run the highest score first. That is the whole system, and its entire value is that multiplication does not have feelings.

Scoring honestly

• Traffic is a lookup, not an opinion. Pull page sessions from analytics before scoring. Ideas on low-traffic pages are hobby projects wearing lab coats: remember from tip 7, thin traffic means tests that cannot reach a verdict.
• Impact is where your funnel data earns its keep. Ideas aimed at the leaky step you circled in tip 11 score high; ideas polishing an already-healthy step score low, however pretty the mockup.
• Ease includes the undo. A change you can revert in five minutes is worth an extra point. A change that needs a developer and an apology tour is not “a 4.”
• Rescore monthly. Traffic shifts, seasons turn, and last month’s number 3 idea may be this month’s number 1.

Keep the backlog in a plain spreadsheet where everyone can add ideas and see the scores. The transparency is half the value: when the loudest voice in the room can watch their pet idea lose to arithmetic in public, your testing program stops being politics and starts being compounding.

A/B testing on Shopify comes down to three disciplines: change one variable, pick your success metric before you start, and do not peek early. Every failed testing program we have ever seen broke at least one of the three. Usually all three, in the same test, with a celebration afterward.

One variable

If you change the headline, the hero image, and the button color in one go and conversions rise, what did you learn? That “stuff” works. You cannot repeat “stuff.” You cannot roll “stuff” out to other pages. A test that cannot teach you anything is a redesign with extra paperwork. One change per test, however impatient that makes you. The impatience is the point: it forces you to bet on the change you actually believe in.

Pick the metric first

Decide, in writing, before launch: “this test wins if revenue per visitor goes up.” Not after. Here is why the order matters: any test throws off a dozen numbers, and some will be up by pure chance. Choose the metric afterward and you will find a winner every time, because you are shopping for one. That is not measurement, that is astrology with dashboards. Write down the metric, the expected direction, and roughly what you will do if it wins or loses. Thirty seconds of writing keeps you honest for two weeks.

Which metric? Revenue per visitor is the safest default, for the reason we covered in tip 1: it catches “wins” that lift conversion while quietly shrinking order size. If the change lives high in the funnel, add-to-cart rate is a fair leading indicator. But name it before the traffic flows.

Do not peek

Peeking is checking results daily and stopping the moment your variant is ahead. It feels diligent. It is the single most reliable way to crown a fake winner, because early results swing wildly on small numbers, and if you keep checking, you will eventually catch a random high and call it victory. The fix is boring: decide the sample size or duration up front, then leave it alone. Evan Miller’s sample size calculator tells you how many visitors you need; run it before launch and let the answer humble you. Whatever the math says, run whole weeks, because weekend shoppers and Tuesday-lunch shoppers are different people, and a test that ends Friday only surveyed half your audience.

And if the calculator says you need three months of traffic to detect the lift you are hoping for? Good, now you know, and you can test bigger swings instead: a different offer, a restructured page, a new hero concept. Small stores earn real results by testing boldly, not by measuring ripples with a ruler.

Testing is a skill we will build on all the way to November: what to test first, when a test is a waste of traffic, and when to shut the lab entirely. Today, just internalize the three disciplines. They are free, and they are rarer than any tool.