/Docs
ETEN

Lift and incrementality

What it is

The console shows two "impact" numbers of different strength:

  • Descriptive lift — the share of engine-associated purchases compared to control purchases. Shows association, not causation.
  • Incrementality (holdout) — the result of a randomised experiment: part of the audience (the holdout group) is deliberately kept on generic content, and revenue per recipient (RPP) is compared between the two arms. This is the only number that proves the engine created additional revenue.

How it is computed

  • Descriptive lift: the day's engine-purchase rate vs control-purchase rate, from the same attribution classification.
  • Incrementality: customers were randomly split into treatment and holdout arms. RPP is computed for each arm — the arm's total revenue divided by ALL members of the arm, including those who bought nothing. A 95% confidence interval is computed for the difference. Until the interval excludes zero, the panel explicitly says "collecting data" instead of a final number.

How to interpret it

  • Descriptive lift can move for reasons unrelated to the engine (number of campaigns, seasonality, plugin signal coverage). Use it to watch the trend, not to report a result.
  • Quote the incrementality number only once the confidence interval excludes zero. With a small email list this typically takes 8-12 weeks. An early number with a wide interval is directional, not proof.
  • RPP counts non-buyers as zeros — deliberately, because "didn't buy" is also an outcome.

Who sees it

The holdout numbers only reach the console once an operator manually flips a per-tenant visibility switch — never automatically, and never before the 95% CI excludes zero. The experiment's existence is not hidden, though: while a holdout is running but the switch is still off, the console shows a banner explaining that a controlled measurement is in progress, what it will prove once ready, and a rough "results expected in ~N weeks" estimate (the same confidence-interval shrinkage the admin panel uses, falling back to the general 8-12 week range early on). An account where a holdout has never been enabled sees no lift panel at all — it gets programme reach & response instead (next section).

When the holdout is switched off

Ending a measurement does not erase its result. When the holdout is switched off (for example, moving to 100% personalisation), the end of the measurement window is recorded and the last result stays visible in the console and the admin panel as a finished measurement: the same numbers, plus the period they cover (start … end) and a note that the holdout is now off. The calculation stops at the end of that window — once the holdout is off, the former holdout group starts receiving recommendations again, so later orders no longer enter the comparison.

In that state the result is about that period, not about today: nothing claims a control group is currently running. Switching the holdout back on resumes the same experiment (same start date) and clears the end marker.

When there is no holdout at all

A holdout is an option you choose at onboarding, not a rule. An account that runs 100% personalisation never produces an incrementality number, so instead of a permanently empty lift panel the console shows Programme reach & response — on Home as a Personalisation reach tile, on Insights as a card in the same slot. It is ordered by how much each number can be trusted:

  1. Reach and fill quality — how many contacts on your Smaily list carry live picks, how many of the 9 slots are filled on average, how many products are in rotation out of your recommendable catalogue, and when the picks were last refreshed. These are verifiable facts: you can open a contact in Smaily and see the same fields.
  2. How personal the picks are today — the share of slots chosen by a behavioural trigger, by segment best-of, or by best-sellers (cold start). A large best-seller share is normal early on and shrinks as order history accumulates.
  3. Credited revenue since activation — the same attribution number as everywhere else in the console, counted from the day the account was switched on. It is credit, not proof, and while it is still zero it says so in words instead of showing a zero.

Nothing on this card is causal, and none of it is worded as lift. The card states once, as a footnote, that no control group is running for the account — and that a time-boxed measurement can be started whenever you want one. Doing so takes effect the same day: the reach card is replaced by the measuring panel, and the numbers above keep working unchanged.

"Since activation" is the day an operator activated the account (recorded in the admin audit log). For accounts activated before that was recorded, the console counts from the first recommendations issued and says so.

Common misreadings

  • "Lift is 40%, can I tell my client/management +40%?" — not from descriptive lift, and not before the holdout interval excludes zero.
  • "Why do you deliberately hold customers out?" — the holdout is the only honest way to separate what the engine created from what it merely took credit for. Without it, any lift figure would be contestable.
  • "The confidence interval is wide — is something broken?" — no, that is small-sample mathematics. The interval narrows as data accumulates.
  • "The Insights chart compares engine against a control group" — it does not. The second line is "no touch detected": purchases where we found no recommendation touch. Detection depends on tracking, so a missing signal moves a purchase into that line. It is descriptive context, never a lift claim.

Technical background

Descriptive lift: lib/metrics/compute-daily.ts (lift_pct = engine rate / control rate − 1). Incrementality: the Health "Incrementality (holdout)" panel (components/dashboard/widgets/HoldoutLift.tsx) — treatment vs holdout RPP, buyer-rate significance, an AOV/frequency decomposition, and the readiness estimate, all in one panel, gated on the 95% CI; methodology study in docs/RESEARCH_incrementality_measurement.md. The console-visibility gate is tenant_settings.incrementality_client_visible; lib/story/loader.ts only exposes the real figures (Story.lift.incrementality) once it is on — while off, it still exposes non-numeric experiment status (Story.lift.experiment: active flag, start date, readiness weeks) so the console banner (components/console/insights-sections.tsx's LiftCard) can describe the running measurement without leaking any number.

Programme reach & response: lib/story/reach.ts (loadProgrammeReach — live picks per synced contact, slot-source mix, recommendable catalogue, attributed revenue since the activation anchor from admin_audit_log with a MIN(recommendations.issued_at) fallback) rendered by ProgrammeReachCard in components/console/insights-sections.tsx. It is loaded and shown only when Story.lift.display === 'none', i.e. no measurement exists — there is no separate per-tenant setting. Design study: docs/RESEARCH_console_no_holdout.md.

Last updated: 2026-08-06