A/B tests started after mid-October are running on borrowed truth. This is last call: any big test you want to trust through Q4 needs to start this week, because the traffic that will flow through it from here on is progressively less like the traffic your store lives on the rest of the year.
The contamination is not one event, it is a slow tide. From late October your mix shifts: more gift buyers shopping for someone else, more first-time visitors from holiday ads, more deal-hunters comparison shopping across ten tabs, more mobile sessions squeezed into commutes and couches. These people click differently, carry different intent, and respond to different persuasion than your September regulars. A variant that wins with them has proven it wins with them, in a mix that exists six weeks a year.
The testing calendar, honestly drawn
What a test can tell you depends on when its traffic flowed, not on how well you ran it.
A two-week test started today still finishes inside the clean window, with a buffer to roll the winner out calmly. A test started November 1 finishes as a study of deal-hunters.
Triage your test backlog today
- Start now: anything big, structural, and reversible. Layout changes, PDP restructures, the sort-order test from tip 49. Two full weekly cycles still fit before the tide comes in.
- Park until January: anything you cannot start this week. An honest backlog beats a contaminated result, and January traffic is the closest thing to a control group you get all winter.
- Exempt from all of this: obvious fixes and broken things. Those are not tests, just ship them (that argument gets its own tip later in the series).
The deeper point: a test is a measurement instrument, and instruments are only valid in the conditions they were calibrated for. From here to Cyber Monday your store’s conditions change weekly, which is why the series keeps splitting testing into two seasons: the lab season, which is ending, and the playbook season, which is about to begin. Tip 80 marks the full freeze. This week is the last comfortable boarding call before it.
