What is an incrementality test?

An incrementality test, also called a lift test, creates the comparison that observational data cannot: what happens with the marketing, against what happens without it, everything else equal. The difference is the incremental effect of that activity in that period. Because the comparison is controlled, the reading is causal, which is why test results are the gold-standard input to a model.

A geo experiment splits a market into regions, runs the activity in some and not in others, and compares. A conversion lift study, run by a platform such as Meta, holds back the ad from a random share of the audience and compares conversions. A pulse test varies the spend in one channel over time, on and off or at different levels, and reads the response. Each needs enough scale to show an effect above the noise, and each measures one channel in one period.

A test result is a point estimate with uncertainty, and that is exactly what a Bayesian prior is. The result goes into the model as the starting belief about that channel's return, and the model still estimates the final answer across all channels and the whole period. There is no need to run a separate reconciliation between the test, the model and attribution; the integration happens inside one model.

Odins designs tests for channels where the model's range is too wide to act on: one channel, one change at a time, pulsed, with the cost of the test against the optimal plan stated up front. It also takes a customer's existing lift and geo results into the model, and media platforms often co-fund a test when they know an MMM is measuring it.

• Geo experiment: Run the activity in some regions and not others, then compare. The classic causal test, when the market is big enough to split.
• Pulse test: Vary one channel's spend over time, on and off or up and down, so the model can see how it responds.
• Calibration: Anchoring a model to outside evidence, usually experiment results. In a Bayesian model, calibration is a prior.
• Incremental sales: The sales a channel caused that would not have happened without it. The only kind of sales worth paying for.
• Triangulation: Combining MMM, attribution and experiments. The question is where: inside one model, or in a dashboard afterwards.
• Prior: What the model believes about a channel before it sees the data. The part of MMM that decides how much the data can teach.

Go one level deeper: Recommendations and testing. Triangulation: MMM, attribution and experiments. Or go back to all terms in the glossary.