Holdout Test: How Ecommerce Brands Measure What Their Ads Actually Cause

A holdout test measures what advertising causes by withholding ads from a control group of people or regions and comparing their purchases with a group that saw the ads. The difference is the incremental effect. It is the only method that separates sales the ads created from sales that would have happened anyway.

What is a holdout test?

Every attribution model, from last click to the most sophisticated multi-touch tool, assigns credit for sales that already happened. A holdout test asks a different question: what would have happened without the ads? To answer it, a portion of the audience is deliberately not shown the campaign for a set period. If the exposed group buys more than the holdout group, the gap is the lift the ads produced, and dividing the extra revenue by the spend gives an incremental return on ad spend. Google’s documentation for its Conversion Lift product describes exactly this structure, a treatment group that sees the ads and a control group that does not, and reports incremental conversions, incremental conversion value, incremental cost per action and incremental ROAS. The method is old; what is new for ecommerce brands is that the platforms now offer it directly, and that the gap between platform-reported ROAS and incremental ROAS has become too large to ignore when budgets are set from the former.

User holdout or geo holdout: which should an ecommerce brand run?

There are two designs, and Google’s own comparison of its two Conversion Lift types lays out the trade. A user-based holdout splits individual people into test and control, which allows reporting by age, gender and conversion type, but Google notes it relies largely on opt-in traffic and can be affected by measurement gaps. A geography-based holdout splits regions instead, which Google describes as noisier and impossible to slice by demographics, but it supports offline sales and cross-channel measurement and does not depend on tracking individuals. For a Shopify brand running Meta and Google together, the geo design usually answers the question that matters, which is whether the whole paid program is adding sales, because it can be read from store orders by region rather than from any platform’s pixel. Both Google designs require working with a Google account representative, and Google’s page says the geography version tends to need a larger budget. Meta offers its own lift studies through its sales channel. Independent tools run geo tests on store data without either platform’s involvement.

What do you need in place before a holdout test is worth running?

Three things, in order. First, clean tracking, because a test compares store outcomes and the store has to record them correctly: orders, new versus returning customers, and region. Second, enough volume. A holdout needs a difference large enough to detect, and a brand spending a few thousand dollars a month will rarely see a statistically clear result; the tests that produce clean answers tend to come from accounts spending $50K a month or more, where even a modest lift translates into hundreds of orders. Third, the willingness to lose some sales for a few weeks, since withholding ads from a control group costs real revenue during the test. The tracking layer is where most tests fail before they start, which is why the measurement work at plaidtesting.com/services/tracking-attribution begins with reconciling what the store recorded against what each platform claimed, so that any later holdout compares real numbers rather than two different estimates of the same sale.

How do you read the result and change the budget?

Read the incremental return, not the platform’s. If Meta reports a 4.0 ROAS and the holdout shows an incremental ROAS of 1.8, both numbers are true in their own terms, and only the second one tells you what another dollar of spend will buy. Compare the incremental figure with the break-even you set from contribution margin: above it, the channel can take more budget; below it, spend is partly buying customers who were coming anyway. Then retest, because incrementality changes with season, creative and competition, and a result from March says little about November. Treat holdout results as calibration for the everyday numbers rather than a replacement for them, since no brand can run a holdout every week. For the everyday layer, from attribution windows to new-customer reporting, this guide to ecommerce attribution covers what each method can and cannot tell you.

An athletic apparel brand I work with shows the question a holdout exists to answer. In June 2026 its returning customer revenue reached $759.5K, up from $531.3K in June 2025, a 42.9 percent increase, according to the brand’s Triple Whale reporting. Retargeting and returning-customer campaigns ran throughout, and the platforms claimed credit for much of that revenue. How much of it those campaigns caused, as opposed to customers who would have reordered anyway, is precisely what no attribution report can say and what a holdout would have measured. The revenue figure is the store’s total across all channels, nothing was withheld from any group during that month, and product launches prompted reorders alongside the ads. The honest reading is that it is a strong result with an unmeasured share.

Before your next budget increase, pick the one campaign you are least sure is incremental and plan a four-week geo holdout on it.