How accurate is marketing mix modelling, and how do you check a vendor's model?

Every marketing mix modelling vendor will tell you their model is accurate. Most will show you a chart where the model's line sits on top of your historical sales. That chart proves less than it looks. This guide explains what accuracy means for a marketing mix model, which tests separate a model that predicts from one that merely explains, how to read the accuracy number a vendor gives you, and what to ask before you trust a model with a budget.

The short answer

Why fit is the easy half

A marketing mix model has many moving parts: a response curve per channel, carryover, a baseline, seasonality, controls for price and promotions. With that many parameters, fitting two years of weekly sales is not hard. A model that has learnt nothing true about your channels can still trace the history closely, because it has enough freedom to explain the past by whatever means fit.

That is why "R-squared of 0.95" or "the model explains 97 percent of the variance" tells you almost nothing about whether the model is right. It tells you the model is flexible. The test that matters is the one the model cannot pass by flexibility alone: predicting sales it has not seen.

The three tests that mean something

1. Stored forecasts against actuals

This is the test to insist on. The model is trained up to a cutoff date. Its forecast for the following weeks is stored at that point, before the sales come in. When the actuals arrive, they are laid next to the forecast, and the gap is reported. Nothing is adjusted afterwards.

The word "stored" carries the weight. A forecast produced after the fact, on a model that has since been retrained on the same period, is a fit dressed up as a forecast. Ask whether the comparison is made in the product, every month, without the vendor preparing it, and whether you can see it for the model you were using three or six months ago. If every older model is kept, you can ask how well the model from last spring predicted the autumn. If older models are overwritten, nobody can check.

2. A holdout the model never saw

Before you have months of live forecasts, you can run the same test backwards. Train the model on data up to a date in the past, say twelve months ago, and let it forecast the following quarter. You know what happened in that quarter; the model does not. A vendor who is confident in their method will run this for you on your own data during a pilot. The result is a preview of what the monthly verification will look like once you are live.

One caveat: a holdout is a single period. A quarter with a tracking outage or a competitor's launch will look worse than the model deserves. Read it next to the month-by-month record once that exists.

3. Did the data teach the model anything?

A Bayesian model starts from priors: what the vendor and your team believed about each channel before the model saw your data. After training, every channel has a posterior, which is what the model believes now. Ask to see the two next to each other. Where the posterior has moved and narrowed, the data had something to say. Where the posterior sits on top of the prior with the same width, the data was silent and you are looking at an assumption, not a finding.

That is normal for some channels. A channel whose spend has never varied, or a new one with three months of history, cannot be identified from the data alone, and a good vendor will say so. The red flag is a vendor who cannot show you the priors at all, or who presents every channel result with the same confidence.

How to read the accuracy number

When a vendor says "92 percent accurate", four things decide whether that number means what it sounds like.

• How it is calculated. Usually the gap between forecast and actual sales, as a share of actual sales, averaged over the period. A gap measured week by week is stricter than one measured on the quarter's total, because weekly errors in opposite directions cancel out in the total. Ask which one the number is.
• Over what horizon. Four weeks ahead is easier than thirteen. Ask how far ahead the forecast ran and whether the model was retrained in between.
• On which model. The fair test is the model you were actually using during the period, from its training cutoff to the next model's cutoff. A model retrained on the period and then "verified" on it is back to fit.
• On what period. A stable quarter flatters any model. A period with a product launch, a price change or a tracking bug will show a wider gap, and the right response from the vendor is to show it and explain it, not to drop the month.

What is a good number? For a model in steady state, with a year or more of monthly retraining behind it, forecasts within 10 to 15 percent of actual sales on a weekly comparison is what we aim for. A first model typically runs wider while it learns the customer's patterns, and narrows over the first few months. A vendor who quotes a tighter number without saying how it was measured has given you a marketing figure.

An accurate forecast can still be wrong about channels

This is the part most buyers miss. Total sales can be forecast well by a model that has the split between channels wrong. The baseline can absorb effect that belongs to brand advertising; two channels that always run together can have their effects swapped; a channel that never varied can carry whatever the prior said. The forecast looks fine. The budget recommendation built on it is not.

So forecast accuracy is necessary, not sufficient. Add three checks on the inside of the model:

• Ranges on every number. Each channel's return should come with an uncertainty range, and the ranges should differ. A channel with three years of varied spend should be tight; a new channel should be wide. Identical, narrow ranges everywhere mean the uncertainty is being hidden.
• The shape of the response curves. Look at the saturation curve per channel and the range of spend you have actually run. Inside that range the curve is estimated; outside it, it is extrapolated from the prior. A recommendation to triple a channel's budget rests on the extrapolated part, and the vendor should say so.
• Agreement with tests you have run. If you have run a geo experiment or a lift test on a channel, the model's estimate for that channel should sit within the test's range. Where it does not, one of the two is wrong, and that is worth a conversation before any money moves. The better vendors take experiment results into the model as priors, so the agreement is built in rather than checked afterwards. How that works is in our guide to triangulation in marketing measurement.

What makes a model less accurate

Some of the gap between forecast and actuals is the model's fault. Much of it is not, and knowing the difference helps you read a verification chart fairly.

• Events the model could not know about. A competitor's launch, a price change that was not in the plan, a stock-out, a tracking bug that drops a week of sales. A model forecasts from what it has seen; it does not foresee. What you can expect is that the gap is flagged quickly, explained, and that known shocks are marked in the history so they do not distort what the model learns next.
• Thin or flat data. A channel whose spend never changed gives the model nothing to learn from. The posterior stays on the prior, the range stays wide, and the forecast for that channel is an assumption. The fix is variation: a structured test that moves the spend enough for the effect to show.
• A new channel. Three months of a new channel is not enough history. A sound model gives it a wide range and treats it as a test until the range narrows.
• Retraining too often. It sounds like diligence to retrain weekly. On weekly data it mostly fits noise, and the channel estimates jump from one week to the next. Monthly retraining on continuously updated data is as often as a budget decision changes, and gives a model that moves slowly and can be verified.

Questions to ask a vendor about accuracy

Take the right-hand column into the meeting. It is the one that is hard to fake.

How Odins checks its own models

Every forecast in Odins is made before the actuals arrive and stored. The verification view shows forecast and actuals side by side, week by week, for every model, including the ones you used months ago. The model is retrained monthly on data that flows in daily, and after each retrain the model you used the previous month is verified over the weeks it was in use. When the forecast and the actuals diverge, the platform flags it and an analyst looks at why before any recommendation is sent. In steady state we aim for forecasts within 10 to 15 percent of actual sales, and we show the months where an external event made the gap wider rather than drop them. The full cadence is on how the model stays accurate, and the method behind the priors and ranges is in our Bayesian approach.

Frequently asked questions

How accurate is marketing mix modelling really?

It depends on what you measure. Any marketing mix model fits its own history well, so fit statistics such as R-squared say little. The meaningful number is forecast accuracy: how close stored forecasts came to the sales that then arrived. For a model in steady state, with monthly retraining and a year or more of history behind it, within 10 to 15 percent of actual weekly sales is a reasonable expectation. A first model is wider.

Is 90 percent accuracy good?

Only if you know how it was measured: on a stored forecast rather than a fit, over what horizon, on which model, and whether any months were left out. A 90 percent figure on weekly actuals over a quarter, from the model that was actually in use, is a strong result. The same figure on the quarter's total, or on a model retrained on the period, is not comparable.

What is a holdout test in MMM?

The model is trained on data up to a past date and asked to forecast the period after it, which you already know. Because the model never saw that period, the gap between its forecast and the actual sales is an honest measure of how well it predicts. It is the same test as monthly verification, run backwards on history, and a fair thing to ask for during a pilot.

Can a model forecast sales well and still be wrong about which channels work?

Yes. Total sales can be predicted well while the split between channels is wrong, for instance when two channels always run together or when the baseline absorbs effect from brand advertising. That is why forecast accuracy is checked together with the ranges on each channel, the shape of the response curves against the spend you have actually run, and agreement with any experiments you have done.

How often should accuracy be checked?

Every month, on the model that was in use during that month, from its training cutoff to the next model's cutoff. Checking less often lets drift go unnoticed; checking the newly retrained model on the weeks it has already seen is fit, not verification.

For the full background on how the method works, read our complete guide to marketing mix modelling. If you are comparing vendors now, how to choose a marketing mix modelling vendor has the wider checklist, and you can book a verification walkthrough on a live model.