Advanced · Lesson 1 of 4
Incrementality: is this channel adding sales?
About 12 minutes · Yogender Kumar, Managing Partner
By the end of this lesson you'll be able to
- Explain the difference between attributed and incremental sales
- Design a holdout or regional test for a channel
- Calculate the cost per incremental sale from test results
Why this matters
Every platform reports the sales it touched. Some of those people would have bought anyway. Until you test, you don't know whether a channel is adding sales or just standing near them.
The core idea
An attributed sale is one a platform claims, because its ad was seen or clicked before the purchase. An incremental sale is one that wouldn't have happened without the ad.
The gap between them can be large, and it's usually largest for channels that reach people already close to buying: retargeting, brand search, and voucher sites.
The way to measure it is an experiment. Split people, or places, into two groups at random. One sees the ads. The other, the holdout, doesn't. Compare how often each group buys. The difference is what the ads added.
There are two common designs:
- Audience holdout: the platform keeps a random share of your audience from seeing ads. Meta and Google both offer tests like this, though availability and minimum spend vary. They measure the platform's own conversions, so a check against your order system makes them stronger.
- Regional test: ads run in some regions and not in matched others. Track 4 used this for connected TV.
A holdout test in two lanes
Test group: sees ads
Holdout: sees no ads
Worked example
Penrose Pets is an imaginary UK pet supplies shop, so the figures are illustrative. It spends £10,000 a month on Meta. Meta reports 500 sales, which is £20 a sale.
Penrose runs a four-week holdout test. Meta keeps 10% of the audience from seeing the ads:
- Test group: 90,000 people saw ads. 1.2% bought: 1,080 buyers.
- Holdout: 10,000 people saw none. 0.9% bought: 90 buyers.
If the test group had seen no ads, it would have bought at the holdout's rate: 90,000 × 0.9% = 810. So the ads added 1,080 − 810 = 270 sales.
Each incremental sale cost £10,000 ÷ 270 = £37.04. That's nearly twice the £20 Meta reported. Penrose's limit is £40, so Meta probably earns its place, but not with the room it looked like.
The honest limits. A difference this size is unlikely to be chance, so the ads did add sales. How many is much less certain. The holdout has only 90 buyers, so a fair range for the cost per incremental sale is about £23 to £97, either side of the £40 limit. Penrose keeps Meta at its current budget rather than raising it, and reruns the test with a bigger holdout.
One test also covers one season, one offer and one set of creative. Penrose plans to repeat it twice a year.
My view: test the channels that look best first. They're usually the ones taking credit for sales that were happening anyway.
Penrose's Meta cost per sale: reported against tested
In practice
- Pick one channel and write down the measure before the test starts.
- Split at random, by person or by matched region.
- Keep the holdout big enough to give a clear answer. Check the platform's power estimate if it offers one.
- Don't change budgets or creative during the test.
- Work out cost per incremental sale, and compare it with your limit.
- Repeat tests once or twice a year.
Common mistakes
- Taking platform numbers as incremental. They count the sales they touched. Test before you trust.
- Holdouts too small to read. A test that can't tell the groups apart wastes a month. Size it first.
- Treating one test as the final word. Results move with season and offer. Repeat it.
Try it yourself
A retargeting test runs for four weeks. 50,000 people see ads and 2% buy. A holdout of 5,000 people sees none and 1.6% buy. Spend was £6,000. How many incremental sales were there, and what did each cost?
Show answer
Test group buyers: 50,000 × 2% = 1,000. Without ads, at the holdout's rate: 50,000 × 1.6% = 800. Incremental sales: 1,000 − 800 = 200. Cost per incremental sale: £6,000 ÷ 200 = £30. The holdout has only 80 buyers, so treat £30 as a rough figure, not an exact one.
Quick check
Not scored. Pick an answer to see the explanation.
What is an incremental sale?
A test group of 40,000 buys at 1.5%. The holdout buys at 1.0%. How many sales did the ads add?
Key takeaways
- Attributed sales are claimed. Incremental sales are added.
- A random holdout shows what would have happened without the ads.
- Cost per incremental sale is spend divided by sales added.
- One test is one season and one offer. Repeat it.