Every marketing mix modelling vendor will tell you their model is accurate. Most will show you a chart where the model's line sits on top of your historical sales. That chart proves less than it looks. This guide explains what accuracy means for a marketing mix model, which tests separate a model that predicts from one that merely explains, how to read the accuracy number a vendor gives you, and what to ask before you trust a model with a budget.
A marketing mix model is accurate when it predicts sales it had not seen when the prediction was made, and keeps doing so month after month. How well it fits its own history says little. Ask the vendor for forecasts that were stored before the actuals arrived, laid next to what then happened, week by week. In steady state, a well-built model lands within roughly 10 to 15 percent of actual sales; a first model is usually wider. Then check the parts: whether the data moved the model away from its starting assumptions, whether each number comes with a range, and whether the channel results hold up against any tests you have run.
A marketing mix model has many moving parts: a response curve per channel, carryover, a baseline, seasonality, controls for price and promotions. With that many parameters, fitting two years of weekly sales is not hard. A model that has learnt nothing true about your channels can still trace the history closely, because it has enough freedom to explain the past by whatever means fit.
That is why "R-squared of 0.95" or "the model explains 97 percent of the variance" tells you almost nothing about whether the model is right. It tells you the model is flexible. The test that matters is the one the model cannot pass by flexibility alone: predicting sales it has not seen.
Two words to keep apart
"Fit" is how well the model explains the past it was trained on. "Forecast accuracy" is how close it came to sales it had not seen when the forecast was made. Vendors who only talk about fit are showing you the half that any model passes.
This is the test to insist on. The model is trained up to a cutoff date. Its forecast for the following weeks is stored at that point, before the sales come in. When the actuals arrive, they are laid next to the forecast, and the gap is reported. Nothing is adjusted afterwards.
The word "stored" carries the weight. A forecast produced after the fact, on a model that has since been retrained on the same period, is a fit dressed up as a forecast. Ask whether the comparison is made in the product, every month, without the vendor preparing it, and whether you can see it for the model you were using three or six months ago. If every older model is kept, you can ask how well the model from last spring predicted the autumn. If older models are overwritten, nobody can check.
Before you have months of live forecasts, you can run the same test backwards. Train the model on data up to a date in the past, say twelve months ago, and let it forecast the following quarter. You know what happened in that quarter; the model does not. A vendor who is confident in their method will run this for you on your own data during a pilot. The result is a preview of what the monthly verification will look like once you are live.
One caveat: a holdout is a single period. A quarter with a tracking outage or a competitor's launch will look worse than the model deserves. Read it next to the month-by-month record once that exists.
A Bayesian model starts from priors: what the vendor and your team believed about each channel before the model saw your data. After training, every channel has a posterior, which is what the model believes now. Ask to see the two next to each other. Where the posterior has moved and narrowed, the data had something to say. Where the posterior sits on top of the prior with the same width, the data was silent and you are looking at an assumption, not a finding.
That is normal for some channels. A channel whose spend has never varied, or a new one with three months of history, cannot be identified from the data alone, and a good vendor will say so. The red flag is a vendor who cannot show you the priors at all, or who presents every channel result with the same confidence.
When a vendor says "92 percent accurate", four things decide whether that number means what it sounds like.
What is a good number? For a model in steady state, with a year or more of monthly retraining behind it, forecasts within 10 to 15 percent of actual sales on a weekly comparison is what we aim for. A first model typically runs wider while it learns the customer's patterns, and narrows over the first few months. A vendor who quotes a tighter number without saying how it was measured has given you a marketing figure.
This is the part most buyers miss. Total sales can be forecast well by a model that has the split between channels wrong. The baseline can absorb effect that belongs to brand advertising; two channels that always run together can have their effects swapped; a channel that never varied can carry whatever the prior said. The forecast looks fine. The budget recommendation built on it is not.
So forecast accuracy is necessary, not sufficient. Add three checks on the inside of the model:
Some of the gap between forecast and actuals is the model's fault. Much of it is not, and knowing the difference helps you read a verification chart fairly.
Take the right-hand column into the meeting. It is the one that is hard to fake.
| Ask | A good answer |
|---|---|
| How do you show that the model predicts sales it has not seen? | Forecasts stored before the actuals, compared week by week, every month, visible in the product. |
| Can we see that comparison for the model we used three months ago? | Yes. Every older model is kept, and its forecast can be checked against what happened. |
| What accuracy do you expect, and how is it measured? | A range, with the metric and the horizon stated: for instance within 10 to 15 percent of weekly actuals in steady state, wider for a first model. |
| Will you run a holdout on our data before we sign? | Yes: train to a past cutoff, forecast a quarter you already know, show the gap. |
| Can we see the priors next to the posteriors, per channel? | Yes, in the product, with the range on each. And here is a channel where the data moved the estimate, and one where it did not. |
| How do the channel estimates compare with tests we have run? | We take your test results into the model as priors, and here is how the two line up. |
| What happens when the forecast drifts from actuals? | The platform flags it, a person looks at whether it is a data issue or a market change, and no recommendation goes out until that is understood. |
| Which periods did you leave out of the accuracy figure? | None. The months with a shock are shown and explained. |
Every forecast in Odins is made before the actuals arrive and stored. The verification view shows forecast and actuals side by side, week by week, for every model, including the ones you used months ago. The model is retrained monthly on data that flows in daily, and after each retrain the model you used the previous month is verified over the weeks it was in use. When the forecast and the actuals diverge, the platform flags it and an analyst looks at why before any recommendation is sent. In steady state we aim for forecasts within 10 to 15 percent of actual sales, and we show the months where an external event made the gap wider rather than drop them. The full cadence is on how the model stays accurate, and the method behind the priors and ranges is in our Bayesian approach.
It depends on what you measure. Any marketing mix model fits its own history well, so fit statistics such as R-squared say little. The meaningful number is forecast accuracy: how close stored forecasts came to the sales that then arrived. For a model in steady state, with monthly retraining and a year or more of history behind it, within 10 to 15 percent of actual weekly sales is a reasonable expectation. A first model is wider.
Only if you know how it was measured: on a stored forecast rather than a fit, over what horizon, on which model, and whether any months were left out. A 90 percent figure on weekly actuals over a quarter, from the model that was actually in use, is a strong result. The same figure on the quarter's total, or on a model retrained on the period, is not comparable.
The model is trained on data up to a past date and asked to forecast the period after it, which you already know. Because the model never saw that period, the gap between its forecast and the actual sales is an honest measure of how well it predicts. It is the same test as monthly verification, run backwards on history, and a fair thing to ask for during a pilot.
Yes. Total sales can be predicted well while the split between channels is wrong, for instance when two channels always run together or when the baseline absorbs effect from brand advertising. That is why forecast accuracy is checked together with the ranges on each channel, the shape of the response curves against the spend you have actually run, and agreement with any experiments you have done.
Every month, on the model that was in use during that month, from its training cutoff to the next model's cutoff. Checking less often lets drift go unnoticed; checking the newly retrained model on the weeks it has already seen is fit, not verification.
For the full background on how the method works, read our complete guide to marketing mix modelling. If you are comparing vendors now, how to choose a marketing mix modelling vendor has the wider checklist, and you can book a verification walkthrough on a live model.