How to shortlist a conversion rate optimization agency in a week
Five days is enough. Day one, define what you are actually buying. Day two, build a longlist of eight. Day three, disqualify on three specific questions. Day four, take two calls. Day five, ask for a reporting template rather than a case study. Most procurement processes take six weeks and produce a worse decision.
Day one: decide what you are buying
An audit, a programme, or build capacity. These are three different purchases and firms specialise differently.
If you have under roughly a thousand conversions a month, you are buying an audit and research, not a testing retainer. The effects being hunted are small, winners in DRIP’s database produced a median 1.88% conversion uplift, and a programme at low volume will spend six months producing inconclusive results.
Write one sentence describing the purchase before you contact anyone. It prevents the common outcome where four agencies pitch four different things and comparison becomes impossible.
Day two: build a longlist of eight
Sources that work: directories with real profiles, the partner directories of tools you already use, and referrals from brands at your stage rather than your aspiration.
Partner directories are underrated. An agency listed with your testing tool or subscription platform has demonstrated actual competence with the stack you run, which is more information than any awards page.
Day three: disqualify on three questions
Send the same three by email. You are looking for specificity, not eloquence.
How do you define a win? There is no standard, which is why it reveals things. ConversionTeam published its results under every definition in use: 50.5% raw winners, 19.1% statistically significant per test, 63.7% decisive once inconclusive tests were excluded. A firm quoting a rate without a definition is quoting the most flattering one.
When does the first test go live? A date, not a phase. The median test runs 42 days, so a six-week discovery means no evidence until month three.
What happens to losing tests? Most results lose or land inconclusive at win rates near 12% across 127,000 experiments. If losses are not reported in the same detail as wins, you are buying a selection.
Three vague answers and you can drop a firm without a call. This usually removes half the longlist.
Day four: two calls, same agenda
| Ask | Listening for |
|---|---|
Which clients do you turn down? | A specific answer. “None” means capacity, not methodology. |
Who will do the work? | The same people who are on the call |
What is your primary metric? | Revenue per visitor, not conversion rate alone |
Is post-purchase in scope? | Either answer is fine. Not knowing is not. |
| Who owns the tool accounts? | Yours, with them added as users |
The first row is the most revealing question in the whole process. A firm that has never declined a client has capacity rather than a method.
Day five: ask for the reporting template
Not a case study. The actual monthly report another client receives, anonymised.
Case studies are marketing. Self-reported performance runs high across the industry: the Econsultancy and RedEye survey summarised by Blend put the average reported winner rate at 39% for agencies, well above audited figures.
A report tells you what the engagement actually feels like. If it opens with traffic and impressions, you are buying a marketing report. If it opens with revenue per visitor, tests shipped, and what was learned from the ones that lost, you are buying a programme.
What a good firm will ask you
Watch the direction of questioning on the calls. A firm that spends the hour presenting has a pitch. A firm that spends it asking about your margin, your return rate, your supply constraints and who approves changes is already doing the work.
The best single signal is whether anyone asks what you have already tried and what failed. That question has no marketing value and enormous diagnostic value, which is exactly why the firms worth hiring ask it and the rest do not.
The thing to fix before you hire anyone
Name the internal person who can approve a test without convening a meeting. Not their seniority, their authority.
That single decision predicts engagement success more reliably than agency choice does. An excellent firm on an account where every change needs three approvals will produce roughly a third of the output of a competent firm on an account with one decision-maker.
If you want a reference point for what the answers above should sound like, Parah Group publishes the process, the metric and the scope rather than a claims page.
