← Back to projects
Email A/B variants side by side — B dims as A is highlighted
Email A/B Testing · Kajabi

Email A/B testing as a learning loop for small lists

Kajabi’s most-requested email gap: vary subject, preview, or body; set the win metric and fallback; apply the winner — without overclaiming what a small list can prove.

14 min read
Role
Lead Product Designer
Timeline
Apr – Jun 2024
Team
1 designer · 1 PM · 2 eng

My role & constraints

Lead Product Designer on Email A/B with 1 designer, 1 PM, and 2 eng.

Owned: end-to-end UX, competitive review of 8 email tools, testing and interviews, and A/B patterns later reused for landing pages.

Co-led: scope with PM and eng for broadcast and automation email.

Constraint: ship on the existing editor; a fuller redesign stayed a separate track (Decision 04).

Business context

79% of creators use Kajabi email; 59% send at least weekly. They need to see what worked so the next email can do better — yet 30% of high-earning creators said email falls short, and 29% already pay other tools (Mailchimp, ConvertKit, ActiveCampaign) about $187/mo for more advanced email features.

In the Messaging & CRM strategy I co-authored, established creators who love Kajabi ranked email the #2 area to improve. In the third-party tool survey, 84% who saw the email category expected A/B testing from Kajabi — and it was still missing. So we designed email A/B as a simple way for small lists to learn, not a copy of big email platforms’ statistical testing.

84%

of creators who saw the email category in the third-party tool survey expected A/B from Kajabi

29%

use marketing/funnel tools outside Kajabi

$187

avg monthly spend on those email tools

The problem

Before email A/B testing, creators couldn’t try two versions of an email inside Kajabi before sending to their whole list. Most A/B tools are built for big audiences; most Kajabi creators don’t have that — and many already don’t trust the email numbers they see.

What creators wanted

  • A/B testing built into Kajabi — expected, and still missing.
  • A simple way to learn: try a different subject, preview, or body → see which got more opens or clicks → use the better one → improve the next email. Clear answers (“Did this do better?”).

Why we couldn’t copy other tools

Big email platforms design A/B for marketers with thousands of subscribers. Most Kajabi creators don’t — so we designed for our users (see Insights), not statistical significance theater.

Insights

This came from the Messaging & CRM strategy I co-authored, a survey about tools creators use outside Kajabi, creator interviews, and a review of A/B in 8 email tools.

  1. 01Creators expected A/B testing — and Kajabi still didn’t have it

    84% of creators who saw the email category expected A/B testing from Kajabi. 29% already paid other tools (~$187/mo) for stronger email features; only 32% called A/B their “most important” feature on those tools — so the real gap was catching up inside Kajabi and earning trust, not building a niche power-user feature.

  2. 02Small lists can’t produce statistically confident winners

    Most creators don’t have enough people on their list for a quick, sure winner. Numbers can also disagree (more opens, fewer clicks). Interviews backed a simple job: show what got more engagement — not a stats class.

  3. 03Ambiguous results need creator control, not auto-answers

    Strategy flagged confusing open-vs-click outcomes and low trust in Kajabi analytics. Short tests also favor people who open early — easy to hide behind a confident “winner” label.

  4. 04Creators wanted full tests, not subject-line-only

    The first MVP brief limited v1 to subject + preheader, two versions, and manually sending the winner to everyone else. Strategy and the competitive review pushed further: creators already paid elsewhere to test body/preview, and subject-only wouldn’t close the gap people expected.

  5. 05Waiting on an editor rebuild would have left the gap open

    The email editor was old and fragile, and a redesign looked like a multi-quarter project. Waiting on that rebuild would have left the 84%-expected A/B gap open longer.

Competitive research 8 email service providers

I owned the competitive synthesis — tearing down A/B across Braze, Beehiiv, ConvertKit, HubSpot, Mailchimp, Klaviyo, ActiveCampaign, and Unbounce before locking the PRD. Three implications shaped the product bets:

  • Don’t gate on list size — most “serious” A/B assumes big audiences; educate on what a short window can prove instead (Decision 03).
  • Subject-only isn’t enough — deeper tools put preview/body behind paid tiers (~$187/mo elsewhere); ship subject, preview, or body in v1 at no add-on cost (Decision 02).
  • Honesty over a confident wrong answer — Braze shows when results aren’t decisive and still lets you proceed; creators set metric + fallback (Decision 01).

Full teardown notes

  • Most “serious” A/B assumes big lists — help centers push large audiences and short windows, and open rates look worse than they are if you don’t explain the window. Don’t gate on list size; do educate on what the window can prove.
  • Subject-only is table stakes — lower plans often lock to subject-only; deeper tools put body/preview behind paid tiers creators already buy elsewhere (~$187/mo). Ship subject or preview or body in v1, at no add-on cost.
  • Honesty beats a confident wrong answer — Braze shows non-significance, lets you proceed, and keeps updating after a “winner” — early leaders can lose later. Creators set metric + fallback; system applies after the window (see Decision 01).
  • Explainers and idea prompts — short-window explainers and later coaching prompts, after the core loop is trusted. V1 = trustworthy loop; coaching features follow.

Decisions

01Creators define winning. No fake significance.

Call: Creators set the winning metric (open or click), test duration, and fallback before the test starts. After the window, the higher metric wins and sends to the holdout. If A and B tie, the pre-chosen fallback becomes the winner automatically — same winner UI, no separate “inconclusive” badge.

Why: Same pattern as the Braze teardown: honesty over a confident wrong answer. AI drafts stay a starting point for variants, not a recommended winner. Insights · 02 cover the small-list constraints.

From Insights · 02

02Subject, preview, or body in v1. No add-on tax.

Call: v1 lets creators vary one element per test — subject, preview text, or body — keeping everything else the same, plus duration, as standard features. Side-by-side editing keeps the difference easy to see, so the result teaches something clear.

Why: Creators asked for a full learning loop, not subject-only table stakes. They already pay ~$187/mo elsewhere for that depth. Shipping flexibility at no add-on cost closes the parity gap the survey flagged. Scope change: the first MVP brief limited v1 to subject + preheader; strategy risks and the teardown pulled body into the shipped design direction (see Insights · 01 & 04).

From Insights · 01 · 04

03Safe defaults so a small list still runs a real test

Call: Default split 25% / 25% / 50% holdout, with rounding and guards for tiny recipient sets so creators can’t ship 0/0/100 by accident. Planned in-product copy: short windows only reflect early openers; explain what the window measured — don’t overclaim (see Decision 01). Ties are handled quietly: equal win-metric → the fallback chosen in settings becomes the winner (same winner UI).

Why: Small lists need a real test by default. Ambiguous open-vs-click outcomes and low analytics trust were explicit strategy risks. Explainers belong next to duration and results — not only in a help article.

From Insights · 03

04Ship on the editor we had, not the redesign we wanted

Scoped separately Email editor redesign (newsletter / blog / writing surfaces) · Classic Editor · Automated emails

Call: v1 adds A/B to the existing broadcast email experience on the modern React editor. The fuller editor redesign — and Classic / automated email coverage — wait as their own track.

The tension: Eng pushed back on touching the email editor — outdated and fragile across Rails vs React, a rewrite looked like a multi-quarter sink. I wanted a redesigned editor with more flexible editing and preview so variants could be first-class; that craft bar was right, but pairing a platform rebuild with a net-new learning product would have delayed the 84%-expected gap.

Why this was the right bet: We separated proving the learning model from rebuilding the writing platform. Email A/B shipped on the compose surface we had. The editor redesign was scoped as a separate project so it could be done well — more fluid editing and preview, and reusable beyond email for newsletter, blog, and other text-writing needs across the product. That kept eng risk contained, got creators a trustworthy A/B loop sooner, and still let me define reusable patterns (variants, win metric + fallback, comparative results) that sibling teams used for landing-page A/B.

From Insights · 05

What we built

A learning loop: variants → options → test → winner

Side-by-side editing keeps subject, preview, and body differences understandable. Creators set the win rule and fallback (Decision 01); the system runs the test and shows what worked — without overclaiming.

“I don’t need a stats lecture — I need to know if this subject actually got more people to open.”

— Creator interview, Messaging & CRM research

01 · Variants

I want to create an A/B test from the start — or convert my broadcast email draft into an A/B test.

Why: creators already writing a broadcast email shouldn’t abandon it to test. I owned this entry path: convert the draft into an A/B (or start A/B fresh), then write version A and a B variation. Side-by-side editing keeps subject, preview, and body differences understandable.

Creators pick the one element to test and write its A and B; the rest of the email stays identical across both versions.

02 · Test options: creators define winning

I want to decide how a winner is chosen — and which version wins if A and B tie.

Why: four controls decide how the test runs. Creators also choose when the first test emails go out.

A/B test distribution — set the size of your test group: 25% / 25% / 50% holdout (default) or 10% / 10% / 90%.

Winning metric — open rate or click rate (click only if the email has links); this determines the version sent to remaining recipients.

Test duration — 1–10 hours after the last test email is sent.

Fallback version — Version A or B applied as the winner if A and B tie on the winning metric. Chosen in settings before the test; no separate inconclusive indicator — results show equal metrics and the fallback marked as winner.

Trust copy (planned next to duration / results): short windows mostly reflect early openers; explain what the window measured — don’t overclaim (Decision 01); a tie quietly uses the fallback you already chose.

03 · Results while the test runs

I want to see how my A/B test performed.

Why: while the test is in progress, creators compare A and B side-by-side (opens, clicks, and email previews). The UI stays comparative — planned helper text that early open rate can shift as the window fills (see Decision 01).

Scroll the image, or click to view it larger.

04 · Winner sent to the rest

I want to see how my winning variant performed.

Why: after the window, the winning variant goes to the remaining contacts — including when A and B tied and the pre-chosen fallback was applied as winner (equal metrics; fallback shown as winner, no “inconclusive” label). The report separates the original A/B sample from the winner send so creators can see both what the test measured and how the winner performed at full reach.

Scroll the image, or click to view it larger.

05 · Click report & win by link

I want to see which links got clicks — and pick a winner from a specific CTA, not only total clicks.

Why: total clicks hide which CTA drove action. The pivot added a clearer click report and let creators win by a specific link (e.g. “Buy Now”) when that was the goal of the test.

What's next 2 roadmap bets

Beyond the shipped broadcast email loop, two bets remained on the Messaging & CRM roadmap:

  • Automated email A/B

    A different product track: A/B on automated emails — triggered series where learning compounds across a flow, not a single broadcast email. Same honesty principles; different surface and metrics.

  • Sender & multi-variable tests

    Alternate sender / friendly-from names, and run tests that vary more than one dimension when the broadcast learning loop is trusted.

Outcome

Broadcast email A/B shipped in two releases (v1–v2). After launch, creator interviews changed the roadmap: creators didn’t just want total clicks — they wanted a click report and to pick a winner based on one specific link (like “Buy Now”). We pivoted and shipped that next. The same A/B pattern was later reused for landing pages.

  • Shipped (v1–v2): test subject, preview text, or body; set how the winner is picked and for how long; the winner auto-sends to the rest of the list. After the pivot: click report and win-by-specific-link.
  • Separate track: automated email A/B (for triggered email flows, not one-time broadcasts) is in progress as a different product.
  • Reused elsewhere: landing pages use the same model — compare variants side-by-side, pick a winner, fall back if there’s no clear winner.

Reused on landing pages

I defined the A/B pattern — variants, how a winner is chosen (and a backup on a tie), and clear side-by-side results. Other teams reused it for landing page A/B so they didn’t have to invent the comparison pattern again.

Why it mattered to the business

Email A/B testing was one of the most expected email features — and competitors already had it. Creators were paying about $187/mo for outside tools just to run tests. Building it into Kajabi closed that gap, gave creators a reason to stay, and let them learn what earns opens and clicks without leaving Kajabi.

Metrics we proposed after ship

Not measured yet — baselines from Messaging & CRM research:

  • % of creators using Kajabi email; revenue tied to email
  • Open/click lift on A/B’d vs non-tested emails
  • High-revenue creators unhappy with email (baseline 41%)
  • Spend on outside email tools (baseline ~$187/mo)
  • Creators gaining at least 1 contact / 30 days (baseline 23%)

Reflection

What worked:

  • Studied competitors first. ConvertKit showed subject-only tests were the bare minimum; Braze showed honest “no clear winner yet” results. That made the gap clear before we locked scope.
  • Cut scope to ship. When the editor rewrite stalled, we shipped A/B on the existing editor instead of rebuilding it — so creators could start testing sooner.
  • Listened after launch. Interviews asked for a click report and win-by-specific-link. We changed the roadmap and shipped that next.

What I’d change:

  • Explain results from day one, not later. A short test can make open rates look misleading — the UI should have said so in the first prototype.
  • Use creators’ own words earlier. Interviews told us the problem; support tickets told us the exact words creators use for it.
  • Measure from week one — how many creators start a test and set a win rule — instead of proposing metrics after ship.

Tradeoff I still sit with: we delayed a fuller editor redesign (and Classic email) so broadcast A/B could ship through v2 and the click-report pivot. That was the right call for speed and risk, and other teams still reused the patterns on landing pages. Automated email A/B is now a separate project. Side-by-side editing and preview never got the polish I originally wanted.

What I did beyond design:

  • Co-wrote the Messaging & CRM strategy
  • Co-led scope with PM and engineering for broadcast and automated email
  • Owned the 8-tool teardown that pushed v1 beyond subject-only tests
  • Turned post-launch interviews into the click-report pivot
  • Defined the A/B pattern — variants, win rule, fallback, side-by-side results — that landing-page teams reused

Takeaway: An A/B test should teach you something, not just crown a winner (Decision 01). On a small list — say 800 people — a few extra opens could just be luck. So decide before you send: what counts as success (opens or clicks), and what sends if it’s a tie. Then treat the result as a hint for next time, not final proof.

More projects