/Docs
ETEN

Lift and incrementality

What it is

The console shows two "impact" numbers of different strength:

  • Descriptive lift — the share of engine-associated purchases compared to control purchases. Shows association, not causation.
  • Incrementality (holdout) — the result of a randomised experiment: part of the audience (the holdout group) is deliberately kept on generic content, and revenue per recipient (RPP) is compared between the two arms. This is the only number that proves the engine created additional revenue.

How it is computed

  • Descriptive lift: the day's engine-purchase rate vs control-purchase rate, from the same attribution classification.
  • Incrementality: customers were randomly split into treatment and holdout arms. RPP is computed for each arm — the arm's total revenue divided by ALL members of the arm, including those who bought nothing. A 95% confidence interval is computed for the difference. Until the interval excludes zero, the panel explicitly says "collecting data" instead of a final number.

How to interpret it

  • Descriptive lift can move for reasons unrelated to the engine (number of campaigns, seasonality, plugin signal coverage). Use it to watch the trend, not to report a result.
  • Quote the incrementality number only once the confidence interval excludes zero. With a small email list this typically takes 8-12 weeks. An early number with a wide interval is directional, not proof.
  • RPP counts non-buyers as zeros — deliberately, because "didn't buy" is also an outcome.

Who sees it

The holdout numbers only reach the console once an operator manually flips a per-tenant visibility switch — never automatically, and never before the 95% CI excludes zero. The experiment's existence is not hidden, though: while a holdout is running but the switch is still off, the console shows a banner explaining that a controlled measurement is in progress, what it will prove once ready, and a rough "results expected in ~N weeks" estimate (the same confidence-interval shrinkage the admin panel uses, falling back to the general 8-12 week range early on). Only tenants where a holdout has never been enabled see the plain "numbers here are descriptive" message — saying that while an experiment is actually running would be wrong.

Common misreadings

  • "Lift is 40%, can I tell my client/management +40%?" — not from descriptive lift, and not before the holdout interval excludes zero.
  • "Why do you deliberately hold customers out?" — the holdout is the only honest way to separate what the engine created from what it merely took credit for. Without it, any lift figure would be contestable.
  • "The confidence interval is wide — is something broken?" — no, that is small-sample mathematics. The interval narrows as data accumulates.

Technical background

Descriptive lift: lib/metrics/compute-daily.ts (lift_pct = engine rate / control rate − 1). Incrementality: the Health "Incrementality (holdout)" panel (components/dashboard/widgets/HoldoutLift.tsx) — treatment vs holdout RPP, buyer-rate significance, an AOV/frequency decomposition, and the readiness estimate, all in one panel, gated on the 95% CI; methodology study in docs/RESEARCH_incrementality_measurement.md. The console-visibility gate is tenant_settings.incrementality_client_visible; lib/story/loader.ts only exposes the real figures (Story.lift.incrementality) once it is on — while off, it still exposes non-numeric experiment status (Story.lift.experiment: active flag, start date, readiness weeks) so the console banner (components/console/insights-sections.tsx's LiftCard) can describe the running measurement without leaking any number.

Last updated: 2026-07-21