The monthly report a CRO agency should be sending you
A conversion report should open with cumulative held gain, list every concluded test including the losses, show planned against actual sample size for each, and end with what changed on the roadmap. If it opens with traffic and impressions, you are reading a marketing report that has wandered into the wrong meeting.
Open with cumulative, not with this month
Individual wins are small. Winners in DRIP’s experiment database produced a median conversion uplift of 1.88% and a median revenue per visitor uplift of 2.77%.
Reported month by month, that reads as underwhelming forever. Reported cumulatively, three held gains compounding on the same traffic is a materially different business by month twelve. The format decides whether the programme looks like it is working, and only one of the two is telling the truth about how the value accrues.
Ask for a running total of held gains with the date each one shipped, alongside the current month’s activity.
Every concluded test, including the ones that lost
Most tests do not win. ConversionTeam’s audit of 2,288 tests found 19.1% reached statistical significance per test. A report containing only winners is showing you perhaps one test in five and calling it the programme.
Losses belong in the report with the same detail as wins, because a loss that stopped you rolling out an expensive change is a return. It should say what was believed, what happened, and what the team now believes instead.
Self-reported performance runs high across the industry. The Econsultancy and RedEye survey summarised by Blend put the average reported winner rate at 39% for agencies, well above audited datasets. A report reflecting the real base rate is a sign of honesty rather than weakness.
The three numbers that catch a test called early
| Number | What it reveals |
|---|---|
| Planned sample size | Whether the test was designed or improvised |
| Actual sample size | Whether it was stopped early |
| Planned versus actual runtime | The clearest tell of all |
A test that ran eleven days against a planned twenty-eight was called early, whatever confidence number sits next to it. Every testing tool shows a live result, and stopping when the numbers flatter you inflates false positives substantially.
Requiring planned-versus-actual disclosure on every test is the single most useful clause you can add to a reporting format. It costs nothing and it removes the temptation entirely.
What should not be in it
Traffic and impressions, unless they explain a change in the results. Sessions are the media agency’s metric and they contaminate a conversion conversation.
Screenshots as evidence. A picture of a variant is not a result.
Effect sizes stated as a single number. “A 4% uplift” is less honest than “between 2% and 6%”, and the range is what an experienced reader wants.
Anything you cannot interpret without a call. A report that requires narration is a presentation, and presentations are optimised for how the work feels rather than for what it did.
The section most reports are missing
What changed on the roadmap, and why.
This is the part that proves results are feeding the process rather than being filed. If three tests concluded and the roadmap is identical, either nothing was learned or nothing is being applied. Both are worth a conversation.
It should read plainly: this test lost, so we now believe X, so these two items moved up and this one dropped off entirely.
The metric hierarchy to insist on
Revenue per visitor as primary. Contribution margin on anything touching price, shipping or bundle structure, because a discount lifts conversion and destroys margin and the statistics will be correct in both directions.
Refund rate as a secondary read, since a change that lifts order value and returns has moved money sideways. And a sixty-day window on anything claiming to improve retention, because a fourteen-day read on a billing-cycle question is a number rather than an answer.
At a DTC supplements brand we work with, a shipping threshold test produced two million dollars in profit, and it produced a modest conversion figure. A report using conversion rate as its headline would have buried the most valuable result of the year.
Our reporting format, including planned-versus-actual disclosure on every test, sits under conversion rate optimization company.
