When to stop testing a page and move on

Stop testing a page when three consecutive tests have been inconclusive, when the remaining ideas are cosmetic, when the effects you are hunting are smaller than your sample can detect, or when a different page now carries more traffic. Teams over-test familiar pages because the tooling is already set up, not because the opportunity is still there.

Signal one: three inconclusive results in a row

One inconclusive test is normal. ConversionTeam’s audit of 2,288 tests found 19.1% reached statistical significance per test, so most individual results tell you nothing definitive.

Three in a row on the same page is different. It usually means the remaining variation available on that page produces effects below your detection threshold, and running a fourth will produce a fourth inconclusive.

Before moving on, check the alternative explanation: were all three underpowered by design? If the planned sample was never reachable, the page is not exhausted, the test design was wrong.

Signal two: the ideas left are cosmetic

Look at your backlog for that page. If what remains is button colour, font weight and micro-copy, the page has given up its structural opportunities.

Those changes produce effects far below the median. Winners in DRIP’s experiment database produced a median conversion uplift of 1.88%, and cosmetic changes sit well under that. You would need volume most stores do not have to see them at all.

The backlog is a better diagnostic than the results, because it tells you what is left rather than what has happened.

Signal three: the traffic moved

Pages that mattered last year may not matter now. A collection page carrying 20% of sessions can drop to 5% after a campaign shift or a navigation change.

Re-rank your page inventory quarterly by sessions and by revenue influenced. A 5% lift on a page carrying 2% of traffic is noise; the same lift on your main product template is a quarter’s growth.

This is the most common reason a programme feels slow. Not bad tests, tests on pages that stopped being important.

Signal four: wins shrink when they roll out

A variant wins on half the traffic, ships to everyone, and the gain largely disappears.

That usually means the test ran on the products or segments where the effect was strongest and does not transfer across the catalogue. It is a real finding rather than a failure, and it tells you to test at template level rather than page level from here.

Re-measure a fortnight after every full rollout. Most programmes never do this, which is why held gains and reported gains drift apart over a year.

Where to go next

In order of usual return: offer structure, checkout, post-purchase, then the next-highest-traffic template.

Offer structure first because it changes order composition rather than order count, which produces effects large enough to detect. At a DTC supplements brand we work with, a shipping threshold test produced two million dollars in profit and it touched no page layout at all.

Post-purchase next, because it is almost always untested and an offer after payment cannot cost you the order. Across our accounts, post-purchase upsells add around 10% to average order value.

The discipline that prevents this

Re-rank the roadmap after every concluded test rather than following a document written in month one. Five minutes, once a fortnight, on the top five items only.

A roadmap that never changes is a schedule, and a schedule cannot notice that a page has stopped being worth testing. The re-ranking is what surfaces the four signals above before you have spent another quarter on them.

How the roadmap gets re-ranked and when a surface is retired is set out under A/B testing services for ecommerce.