Losing A/B tests teach more than winning ones, and almost every store throws the lesson in the trash. A winner gets shipped, screenshotted, and mentioned in three meetings. A loser gets quietly rolled back and never spoken of again, which is a shame, because the loser is the one carrying new information: it just told you that something you believed about your customers is false.
A winning test mostly confirms what you already suspected; that is why you ran the variant. A losing test, especially the “obvious” winner that lost, is your customers correcting your model of them. That correction compounds: it does not just kill one variant, it should redirect every future test you were about to build on the same wrong belief. Expert intuition loses more often than experts think. When we compared hand-tuned collection sorting against boring defaults across our own fleet, the hand-tuned versions won in only 48% of stores. A coin flip, from people who know their catalogs cold. Your homepage hunches are not doing better.
The test log: one entry, six lines
Fill this in for every test, wins and losses alike. A spreadsheet row per test is plenty.
Six months of these rows is a private playbook of what your customers actually respond to. No agency can sell you one.
Why the log matters more than the memory
Unlogged losers do not stay dead. Teams re-run failed ideas 18 months later because nobody remembers the funeral, and memory launders every result: “it lost” becomes “the timing was off” becomes “we should try it again for BFCM.” The log is the antidote. It also protects your winners from over-generalization: “free shipping messaging won on the cart page in October” is a fact; “urgency works on our customers” is a myth in the making, and the log keeps the fact-sized version on record.
One habit upgrade while you are at it: write the “belief updated” line for losers with the same energy you would give a win. The test that cost you two weeks and proved your best idea wrong did not fail. It bought you the one thing traffic cannot: a truer model of the people you sell to. That is tuition, and you already paid it. Keep the receipt.
