How to judge a CRO agency at month three

At month three you cannot fairly judge results and you can absolutely judge process. Check five things: is there a ranked roadmap, has it been re-ranked on evidence, has a winner shipped, are losses being reported, and is revenue per visitor in the report. Four out of five is a working engagement. Two or fewer is a decision.

Why results are the wrong test this early

The arithmetic makes month three a bad judging point for outcomes.

The first test should launch inside two weeks. The median test in DRIP’s database ran 42 days, so the first result lands near week eight. Most results are not winners: Optimizely’s analysis of over 127,000 experiments puts the average win rate near 12%. By month three you have perhaps two or three clean reads from a low-hit-rate activity.

Judging an engagement on that sample is like judging a hire on two weeks of output. The signal is there, but it sits in how the work is being done rather than in what it has produced.

The five process checks

One: is there a ranked roadmap? 

Ranked by expected value, meaning traffic on the page multiplied by expected effect multiplied by confidence in the evidence. Not a list of ideas in a slide. Ask to see the ranking logic.

Two: has it been re-ranked? 

Twice by now. A roadmap written in month one and followed unchanged is a schedule. Every result, including the losses, should have changed what the team believes and therefore what sits at the top.

Three: has a winner shipped? 

Not won, shipped. Built properly into the template, QA’d across devices, live for everyone. A winner sitting unbuilt is the most expensive thing in the engagement.

Four: are losses reported? 

With the same detail as wins, and with what they changed about the roadmap. An agency showing only successes is showing you a selection of a low-hit-rate activity.

Five: is revenue per visitor the headline metric? 

Not conversion rate alone. Winners in DRIP’s data produced a median 1.88% conversion lift against a 2.77% revenue per visitor lift, which is how independently the two move.

The two that mean leave

Most failures are recoverable with a direct conversation. Two are not.

Nothing has shipped: If three months have produced no live change, the problem is either that the agency cannot build or that your approval chain is blocking. Both are fixable, and if neither side has named the cause by month three, nobody is going to.

You cannot get a straight answer about a result: If asking what the sample size was, or what the planned runtime was, produces deflection rather than a number, the reporting is presentation rather than measurement. That does not improve with time.

The thing to check about yourself

Before concluding the agency is the problem, check the variable you control.

Has one named person been approving changes without convening a meeting? If every variant waited on three stakeholders, the output you are seeing is the output that structure permits, and changing agency will reproduce it exactly.

Count the days between a variant being ready and going live. If the average is over three, the constraint is internal. On a 42-day test cycle, a five-day approval delay adds roughly 12% to every cycle before anything is built.

What good looks like at month six instead

By then results are a fair test. Expect two to four held gains, a documented understanding of your customers that did not exist before, and revenue per visitor measurably above the baseline you recorded before starting.

You did record that baseline, ideally. If not, record one now, because the month-twelve conversation will otherwise be an argument about memory.

Our month-three review format, and what we expect to be judged on at it, sits under CRO agency services.