Seven questions to ask a CRO agency before you sign
The fastest way to tell a real conversion optimization programme from a redesign wearing a testing tool is to ask how the agency defines a win, what happens to losing tests, and which number they report to your finance team. Vague answers to those three predict a vague engagement. Here are all seven questions.
1. How do you define a win?
There is no industry-standard answer, which is precisely why you should ask. ConversionTeam published its results under every definition the industry uses: 50.5% of its 2,288 audited tests produced a raw winner, 19.1% reached statistical significance per test, and 63.7% was the decisive win rate once inconclusive tests were excluded. Same programme, three very different headline numbers.
An agency quoting a win rate without the definition attached is quoting the most flattering one available.
2. What is your primary metric?
If the answer is conversion rate and nothing else, walk. Conversion rate can be lifted by a discount that destroys margin. Revenue per visitor cannot. In DRIP’s data, winning tests produced a median 1.88% conversion lift and a median 2.77% revenue per visitor lift, which tells you the two numbers move independently often enough to matter.
3. When does the first test go live?
Ask for a date, not a phase. The median test in DRIP’s database ran 42 days. Every week of discovery pushes your first readable result a week further out, so a six-week onboarding means no evidence until month three.
4. What happens to losing tests?
Most tests lose or land inconclusive. That is normal and it is not a failure. What matters is whether losses get reported with the same detail as wins and whether they feed the next hypothesis. An agency that only presents winners is showing you a selection, and you are paying for the whole programme.
5. Which tools do you run, and who owns the accounts?
You want specific names and you want the accounts in your company’s name. A reasonable stack looks something like this.
| Job | Tools you should hear named |
|---|---|
| Testing | Intelligems, Shoplift, VWO, Convert, Nelio |
| Behaviour | Microsoft Clarity, Hotjar, session recordings |
| Analytics | GA4, GTM, server-side where the budget allows |
| Post-purchase | Rebuy or equivalent for upsells and offers |
If the agency will not name tools, or the accounts live under their login, you are renting your own data.
6. Does the scope include post-purchase?
Most agencies stop at the thank-you page. That is where a large share of the available margin sits, in upsells, subscription take rate and repeat purchase. Ask directly whether post-purchase and subscription are in scope or out. Out is a legitimate answer. Not knowing is not.
The follow-up question matters more than the first one. Ask what share of the tests they ran last quarter happened after the payment step. If the honest answer is close to zero, post-purchase is on the proposal rather than in the programme.
7. Who actually does the work?
The people in the pitch and the people on your account should be the same people. Ask who runs the analysis, who writes the hypotheses and who builds the variants, then ask to meet them. Then ask how many other accounts that person carries. Senior attention does not scale past a handful of brands, and a name on a slide is not a commitment of hours.
The answers that should reassure you
Specific dates. A metric that finance recognises. Losses discussed without defensiveness. Named tools. And an honest description of the cases where their approach is the wrong fit, because an agency that has never turned down a client has no methodology, only capacity.
For a sense of what the programme should produce, one supplements brand we worked with moved subscription take rate from 25% to between 55 and 60% over six months, roughly tripling the subscriber base. That result came from a sequence of tests on offer structure and post-purchase flow, not from a redesign. The reason it is worth quoting here is the timeline. Six months, not six weeks, which is the honest shape of this work.
One last check
Ask for the reporting template before you sign. Not a case study, the actual monthly report another client receives. If it opens with traffic and impressions, you are buying a marketing report. If it opens with revenue per visitor, tests shipped, and what was learned from the ones that lost, you are buying a programme.
More on how we approach conversion optimization for DTC brands.
