A/B Testing Email Design in Klaviyo: What to Test, How Long to Wait and How to Read the Result

Most Klaviyo A/B tests are subject line tests. They are quick to set up and the results arrive fast. Design tests are less common, which is a missed opportunity, because the design decides what happens after the open: whether the reader scrolls, clicks and buys. Testing design well takes a little more care, mostly in choosing what to measure.

Here is a practical approach, based on how Klaviyo’s own A/B testing works.

What Klaviyo lets you test

Klaviyo’s help center describes campaign A/B tests for subject lines, message content and send times. For content, its own list of ideas includes the appearance and wording of the call to action, how the template is organized, and images compared with GIFs. It also gives two rules worth keeping: test only one variable at a time, and use only two variations. More variations split your audience into smaller groups and make a clear result less likely.

Pick design changes that could matter

A good design test starts with a question you would act on. Some useful ones:

• Does a single hero product outperform a lifestyle image?
• Does a grid of three products beat a grid of six?
• Does the main button work better above the first product or below it?
• Does a short email with one offer beat a longer one with several?

Avoid tests where you would keep the current design whatever the result. If the answer would not change next week’s template, it is not worth a share of your list.

Measure clicks or orders, not opens

This is the part most design tests get wrong. Klaviyo recommends open rate as the winning metric when you test the subject line, preview text or sender, and click rate when you test email content. A placed order rate option is also available for accounts that track a Placed Order metric.

Opens are a poor measure of design in any case, because readers have not seen the design when they decide to open. Klaviyo also notes that Apple Mail Privacy Protection prefetches its tracking pixel, which can inflate open rates, and that open-rate tests count those automatic opens. For a design test, choose click rate, or placed order rate if your list is large enough to produce meaningful order numbers.

Set the size and the duration

For content tests, Klaviyo lets you set a test size and, when the test size is below 100 percent, a test duration. Its own example: with a 40 percent test size and a six-hour duration, 20 percent of recipients get variation A, 20 percent get variation B, and after six hours the winner goes to everyone else. Klaviyo suggests a test size based on your audience and a duration based on the winning metric.

Clicks and orders take longer to arrive than opens, so give a design test enough time. If your audience is small, consider sending both versions to the whole list and applying the lesson to the next campaign instead.

Read the result properly

Klaviyo reports a win probability for the leading variation. According to its help center, a result is labeled statistically significant when the win probability is above 90 percent and each variation has more than 50 successful deliveries. A result can also be labeled promising, in which case Klaviyo suggests repeating the test, or it can be inconclusive.

The results also show the lift: how much better the leading variation performed. A large lift on a small test deserves a repeat before you trust it, while a modest lift that holds up across several campaigns is often the more useful finding.

Treat a single promising result as a hint, not a rule. Repeat the test on a later campaign before you change every template.

Keep a test log

Write down every test: the question, the two versions, the metric, the result and what you changed afterwards. After a few months the log becomes a set of design rules for your brand, built from your own subscribers rather than from generic advice. Roll each confirmed winner into your templates, so every future campaign starts from what has already worked.

Make testing part of production

Every test needs two finished versions of an email, which doubles the design work at exactly the moment the team is busiest. Brands that want the variations without the extra workload can use a Klaviyo email design service such as Nexmail, which designs and writes campaigns, delivers them into the existing Klaviyo account as production-ready HTML built for Klaviyo’s template engine, and includes unlimited revisions.

Test one thing at a time, measure the action that matters, and let your own results shape the next design.