A/B testing on Shopify comes down to three disciplines: change one variable, pick your success metric before you start, and do not peek early. Every failed testing program we have ever seen broke at least one of the three. Usually all three, in the same test, with a celebration afterward.
One variable
If you change the headline, the hero image, and the button color in one go and conversions rise, what did you learn? That “stuff” works. You cannot repeat “stuff.” You cannot roll “stuff” out to other pages. A test that cannot teach you anything is a redesign with extra paperwork. One change per test, however impatient that makes you. The impatience is the point: it forces you to bet on the change you actually believe in.
Pick the metric first
Decide, in writing, before launch: “this test wins if revenue per visitor goes up.” Not after. Here is why the order matters: any test throws off a dozen numbers, and some will be up by pure chance. Choose the metric afterward and you will find a winner every time, because you are shopping for one. That is not measurement, that is astrology with dashboards. Write down the metric, the expected direction, and roughly what you will do if it wins or loses. Thirty seconds of writing keeps you honest for two weeks.
Which metric? Revenue per visitor is the safest default, for the reason we covered in tip 1: it catches “wins” that lift conversion while quietly shrinking order size. If the change lives high in the funnel, add-to-cart rate is a fair leading indicator. But name it before the traffic flows.
Do not peek
Peeking is checking results daily and stopping the moment your variant is ahead. It feels diligent. It is the single most reliable way to crown a fake winner, because early results swing wildly on small numbers, and if you keep checking, you will eventually catch a random high and call it victory. The fix is boring: decide the sample size or duration up front, then leave it alone. Evan Miller’s sample size calculator tells you how many visitors you need; run it before launch and let the answer humble you. Whatever the math says, run whole weeks, because weekend shoppers and Tuesday-lunch shoppers are different people, and a test that ends Friday only surveyed half your audience.
And if the calculator says you need three months of traffic to detect the lift you are hoping for? Good, now you know, and you can test bigger swings instead: a different offer, a restructured page, a new hero concept. Small stores earn real results by testing boldly, not by measuring ripples with a ruler.
Testing is a skill we will build on all the way to November: what to test first, when a test is a waste of traffic, and when to shut the lab entirely. Today, just internalize the three disciplines. They are free, and they are rarer than any tool.
