A/B tests should run in full weekly cycles, ending on the same weekday they started, and this one scheduling rule quietly invalidates more merchant experiments than any statistics mistake. Your store does not have one audience. It has a Tuesday-lunch-break audience and a Sunday-evening-couch audience, and they are different people buying in different moods with different wallets.
Look at your own analytics by day of week and you will see the fingerprint: weekday traffic that browses and bounces, weekend traffic that settles in and converts, or the exact reverse if you sell office supplies. Payday weeks spike. Sunday evenings buy differently than Friday nights. None of this is noise. It is a weekly rhythm as real as your opening hours used to be.
Now run a test from Wednesday to Monday. Variant A got four weekdays and one weekend; anything that appeals to your weekend buyer is undercounted, anything that suits the weekday skimmer is overcounted. You did not measure “A vs B”, you measured “A on these days vs B on those days” and called it science.
The rules, in full
- Run whole weeks. Seven days, or fourteen, or twenty-one. Start Tuesday, end Tuesday. Every weekday appears the same number of times in both variants.
- Never end on a spike. The urge to call the test the morning after a great day is exactly backwards: a spike is the moment your sample is least representative. If the result is real, it will still be there when the cycle completes.
- Never end on a dip either. Killing a variant during a slow Monday is the same coin flip with sad music.
- Decide duration before you start, like you decided the success metric in tip 7. A stopping rule chosen mid-test is not a rule, it is a mood.
- Quarantine abnormal weeks. A payday-sale week, a press mention, a shipping meltdown: if the week was weird, extend the test by a full cycle rather than reading tea leaves.
This matters double as Cyber Monday approaches, because Q4 compresses everything: traffic climbs week over week, and the temptation to run quick three-day tests grows just as the weekly rhythm gets loudest. Resist. A clean two-week answer in September beats a corrupted three-day answer in November, and September is exactly when you still have time to act on what you learn.
Patience is not the glamorous part of optimization. It is the part that makes the rest of it true.
