A/B tests should run in full weekly cycles, ending on the same weekday they started, and this one scheduling rule quietly invalidates more merchant experiments than any statistics mistake. Your store does not have one audience. It has a Tuesday-lunch-break audience and a Sunday-evening-couch audience, and they are different people buying in different moods with different wallets.

Look at your own analytics by day of week and you will see the fingerprint: weekday traffic that browses and bounces, weekend traffic that settles in and converts, or the exact reverse if you sell office supplies. Payday weeks spike. Sunday evenings buy differently than Friday nights. None of this is noise. It is a weekly rhythm as real as your opening hours used to be.

Now run a test from Wednesday to Monday. Variant A got four weekdays and one weekend; anything that appeals to your weekend buyer is undercounted, anything that suits the weekday skimmer is overcounted. You did not measure “A vs B”, you measured “A on these days vs B on those days” and called it science.

The rules, in full

• Run whole weeks. Seven days, or fourteen, or twenty-one. Start Tuesday, end Tuesday. Every weekday appears the same number of times in both variants.
• Never end on a spike. The urge to call the test the morning after a great day is exactly backwards: a spike is the moment your sample is least representative. If the result is real, it will still be there when the cycle completes.
• Never end on a dip either. Killing a variant during a slow Monday is the same coin flip with sad music.
• Decide duration before you start, like you decided the success metric in tip 7. A stopping rule chosen mid-test is not a rule, it is a mood.
• Quarantine abnormal weeks. A payday-sale week, a press mention, a shipping meltdown: if the week was weird, extend the test by a full cycle rather than reading tea leaves.

This matters double as Cyber Monday approaches, because Q4 compresses everything: traffic climbs week over week, and the temptation to run quick three-day tests grows just as the weekly rhythm gets loudest. Resist. A clean two-week answer in September beats a corrupted three-day answer in November, and September is exactly when you still have time to act on what you learn.

Patience is not the glamorous part of optimization. It is the part that makes the rest of it true.

Prioritize your A/B tests by traffic times impact times ease, not by whoever argues loudest in the meeting. Every store has more test ideas than traffic to run them, which makes prioritization the actual skill. The store that runs its best idea first beats the store that runs its founder’s favorite, every quarter, forever.

The scoring takes ten minutes. For each idea, rate three things from 1 to 5: Traffic, how many sessions actually see the thing you are changing; Impact, how big a behavior change you honestly expect if it wins; Ease, how cheap it is to build and undo. Multiply. Run the highest score first. That is the whole system, and its entire value is that multiplication does not have feelings.

Scoring honestly

• Traffic is a lookup, not an opinion. Pull page sessions from analytics before scoring. Ideas on low-traffic pages are hobby projects wearing lab coats: remember from tip 7, thin traffic means tests that cannot reach a verdict.
• Impact is where your funnel data earns its keep. Ideas aimed at the leaky step you circled in tip 11 score high; ideas polishing an already-healthy step score low, however pretty the mockup.
• Ease includes the undo. A change you can revert in five minutes is worth an extra point. A change that needs a developer and an apology tour is not “a 4.”
• Rescore monthly. Traffic shifts, seasons turn, and last month’s number 3 idea may be this month’s number 1.

Keep the backlog in a plain spreadsheet where everyone can add ideas and see the scores. The transparency is half the value: when the loudest voice in the room can watch their pet idea lose to arithmetic in public, your testing program stops being politics and starts being compounding.

A/B testing on Shopify comes down to three disciplines: change one variable, pick your success metric before you start, and do not peek early. Every failed testing program we have ever seen broke at least one of the three. Usually all three, in the same test, with a celebration afterward.

One variable

If you change the headline, the hero image, and the button color in one go and conversions rise, what did you learn? That “stuff” works. You cannot repeat “stuff.” You cannot roll “stuff” out to other pages. A test that cannot teach you anything is a redesign with extra paperwork. One change per test, however impatient that makes you. The impatience is the point: it forces you to bet on the change you actually believe in.

Pick the metric first

Decide, in writing, before launch: “this test wins if revenue per visitor goes up.” Not after. Here is why the order matters: any test throws off a dozen numbers, and some will be up by pure chance. Choose the metric afterward and you will find a winner every time, because you are shopping for one. That is not measurement, that is astrology with dashboards. Write down the metric, the expected direction, and roughly what you will do if it wins or loses. Thirty seconds of writing keeps you honest for two weeks.

Which metric? Revenue per visitor is the safest default, for the reason we covered in tip 1: it catches “wins” that lift conversion while quietly shrinking order size. If the change lives high in the funnel, add-to-cart rate is a fair leading indicator. But name it before the traffic flows.

Do not peek

Peeking is checking results daily and stopping the moment your variant is ahead. It feels diligent. It is the single most reliable way to crown a fake winner, because early results swing wildly on small numbers, and if you keep checking, you will eventually catch a random high and call it victory. The fix is boring: decide the sample size or duration up front, then leave it alone. Evan Miller’s sample size calculator tells you how many visitors you need; run it before launch and let the answer humble you. Whatever the math says, run whole weeks, because weekend shoppers and Tuesday-lunch shoppers are different people, and a test that ends Friday only surveyed half your audience.

And if the calculator says you need three months of traffic to detect the lift you are hoping for? Good, now you know, and you can test bigger swings instead: a different offer, a restructured page, a new hero concept. Small stores earn real results by testing boldly, not by measuring ripples with a ruler.

Testing is a skill we will build on all the way to November: what to test first, when a test is a waste of traffic, and when to shut the lab entirely. Today, just internalize the three disciplines. They are free, and they are rarer than any tool.