Helping creators learn what works in email — without fake certainty

My role & constraints

Lead Product Designer on Email A/B with 1 designer, 1 PM, and 2 eng.

Owned: end-to-end UX, competitive review of 8 email tools, testing and interviews, and A/B patterns later reused for landing pages.

Co-led: scope with PM and eng for one-off and automation email.

Constraint: ship on the existing editor; a fuller redesign stayed a separate track (Decision 04).

Business context

79% of creators use Kajabi email; 59% send at least weekly. They need to see what worked so the next email can do better — yet 30% of high-earning creators said email falls short, and 29% already pay other tools (Mailchimp, ConvertKit, ActiveCampaign) about $187/mo for more advanced email features.

In the Messaging & CRM strategy I co-authored, established creators who love Kajabi ranked email the #2 area to improve. In the third-party tool survey, 84% who saw the email category expected A/B testing from Kajabi — and it was still missing. So we designed email A/B as a simple way for small lists to learn, not a copy of big email platforms’ statistical testing.

The problem

Before email A/B testing, creators couldn’t try two versions of an email inside Kajabi before sending to their whole list. Most A/B tools are built for big audiences; most Kajabi creators don’t have that — and many already don’t trust the email numbers they see.

What creators wanted

• A/B testing built into Kajabi — expected, and still missing.
• A simple way to learn: try a different subject, preview, or body → see which got more opens or clicks → use the better one → improve the next email. Clear answers (“Did this do better?”).

Why we couldn’t copy other tools

Big email platforms design A/B for marketers with thousands of subscribers. Most Kajabi creators don’t — so we designed for our users (see Insights), not statistical significance theater.

Insights

This came from the Messaging & CRM strategy I co-authored (including risks we planned for), a survey about tools creators use outside Kajabi, creator interviews, and a review of A/B in 8 email tools.

1. 01 · Outside-tools survey · Messaging & CRM appendix

   Creators expected A/B testing — and Kajabi still didn’t have it

   84% of creators who saw the email category expected A/B testing from Kajabi. 29% already paid other tools (~$187/mo) for stronger email features; only 32% called A/B their “most important” feature on those tools — so the real gap was catching up inside Kajabi and earning trust, not building a niche power-user feature.

   Led to Decision 02 — let creators test subject, preview, or body as a normal feature, not subject-line-only behind a paid add-on.

2. 02 · Strategy risks · creator interviews

   Small lists can’t support fake “sure” winners

   Most creators don’t have enough people on their list for a quick, sure winner. Numbers can also disagree (more opens, fewer clicks). Interviews backed a simple job: show what got more engagement — not a stats class.

   Led to Decision 01 — creators pick how a winner is chosen, plus a backup if A and B tie. Same honest approach we saw in the Braze review.

3. 03 · Strategy risks · low trust in analytics

   Messy results need creator control — and clear wording

   Strategy flagged confusing open-vs-click outcomes and low trust in Kajabi analytics. Short tests also favor people who open early — easy to hide behind a confident “winner” label.

   Led to Decision 03 — safe defaults plus planned in-product tips (what the time window measured; don’t overclaim). See The experience · Test options.

4. 04 · Early MVP brief → strategy / competitive review

   Scope grew from subject-only to a fuller test

   The first MVP brief limited v1 to subject + preheader, two versions, and manually sending the winner to everyone else. Strategy and the competitive review pushed further: creators already paid elsewhere to test body/preview, and subject-only wouldn’t close the gap people expected.

   Led to Decision 02 — v1 can vary subject, preview, or body; holdout + backup stay; auto-scheduling the winner moves to a later phase.

5. 05 · Eng constraints · strategy timing

   The learning loop had to ship before an editor rebuild

   The email editor was old and fragile, and a redesign looked like a multi-quarter project. Waiting on that rebuild would have left the 84%-expected A/B gap open longer.

   Led to Decision 04 — ship A/B on the compose surface we already had; scope the fuller editor redesign as a separate track.

Those findings set the product direction before polished screens — then the experience had to show that honesty in the UI, not only in the doc.

The experience

A learning loop: variants → options → test → winner

Shipped UX · v1–v2, pivot One-off email A/B through schedule-send-to-winner; click report / win-by-link from user interviews — screens may be prototype unless noted

Before: one email, one send. No way to compare subject, preview, or body inside Kajabi before committing the whole list.

“I don’t need a stats lecture — I need to know if this subject actually got more people to open.”

Side-by-side editing keeps subject, preview, and body differences understandable. Creators set the win rule and fallback (Decision 01); the system runs the test and shows what worked — without overclaiming.

01 · Entry

Start or convert

New A/B, or turn a broadcast draft into an A/B (and back)

→

02 · Options

Metric · split · duration

Open or click rate, fallback, window, schedule send

→

03 · Test

Two groups

A and B send automatically to the test split

→

04 · Winner

Holdout send

Higher metric wins; on a tie, fallback is applied as winner

01 · Entry: start new, or convert a broadcast

I want to create an A/B test from the start — or convert my email broadcast draft into an email A/B test.

Why: creators already writing a one-off broadcast shouldn’t abandon it to test. I owned this entry path: convert the draft into an A/B (or start A/B fresh), then write version A and a B variation. Side-by-side editing keeps subject, preview, and body differences understandable.

02 · Test options: creators define winning

I want to decide how a winner is chosen — and which version wins if A and B tie.

Why: four controls decide how the test runs. Creators also choose when the first test emails go out.

A/B test distribution — set the size of your test group: 25% / 25% / 50% holdout (default) or 10% / 10% / 90%.

Winning metric — open rate or click rate (click only if the email has links); this determines the version sent to remaining recipients.

Test duration — 1–10 hours after the last test email is sent.

Fallback version — Version A or B applied as the winner if A and B tie on the winning metric. Chosen in settings before the test; no separate inconclusive indicator — results show equal metrics and the fallback marked as winner.

Trust copy (planned next to duration / results): short windows mostly reflect early openers; explain what the window measured — don’t overclaim (Decision 01); a tie quietly uses the fallback you already chose.

03 · Results while the test runs

I want to see how my A/B test performed.

Why: while the test is in progress, creators compare A and B side-by-side (opens, clicks, and email previews). The UI stays comparative — planned helper text that early open rate can shift as the window fills (see Decision 01).

Scroll the image, or click to view it larger.

04 · Winner sent to the rest

I want to see how my winning variant performed.

Why: after the window, the winning variant goes to the remaining contacts — including when A and B tied and the pre-chosen fallback was applied as winner (equal metrics; fallback shown as winner, no “inconclusive” label). The report separates the original A/B sample from the winner send so creators can see both what the test measured and how the winner performed at full reach.

Scroll the image, or click to view it larger.

05 · Click report & win by link Shipped

I want to see which links got clicks — and pick a winner from a specific CTA, not only total clicks.

Why: total clicks hide which CTA drove action. The pivot added a clearer click report and let creators win by a specific link (e.g. “Buy Now”) when that was the goal of the test.

Outcome

One-off email A/B shipped through v2. Interviews drove a pivot — click report and win-by-specific-link. The A/B pattern was reused for landing-page A/B.

• Shipped: subject, preview, or body tests; auto-send the winner; then click-level wins after the pivot. Strategy approved; a dedicated PM unlocked delivery.
• In progress separately: sequence email A/B (different product).
• Systems: same compare / win / fallback model on landing pages.

2024 company goals · How this helped

• Catch up & keep creators (parity & retention): fund a top expected email gap so people can get depth inside Kajabi instead of paying ~$187/mo elsewhere.
• Brand & revenue (GMV): honest A/B for small lists as a way to learn what earns opens and clicks — and better conversions — without “export to Mailchimp to learn.”

Metrics we proposed after ship

Not measured yet — proposed in the strategy (baselines from Messaging & CRM research):

• % of creators using Kajabi email; revenue tied to email
• Open/click lift on A/B’d vs non-tested emails
• High-revenue creators unhappy with email (baseline 41%)
• Spend on outside email tools (baseline ~$187/mo)
• Creators gaining at least 1 contact / 30 days (baseline 23%)