Incrementality testing overview
Incrementality testing measures the causal lift that advertising produces above what would have happened without it. Use this section when you want to know whether your Apple Ads spend is creating new installs, rather than how installs that already happened should be credited. The test requires pausing selected campaigns or keywords for a defined window, so plan for the opportunity cost before you start.
What incrementality is, and what it is not
Incrementality is the causal lift attributable to an intervention. In advertising, the intervention is the ad activity itself. The incremental installs for a campaign are the installs that would not have occurred if the campaign had been off during the measurement window.
Three related terms are often confused.
- Attribution divides credit for observed installs across the touchpoints that preceded them. An install attributed to a campaign may or may not be incremental. Some of those installs would have happened from organic search, competitor conquesting, or other paid channels.
- Correlation is what happened alongside the campaign. Installs rising when spend rises is correlated with the campaign but does not, on its own, show the campaign caused the rise.
- Incrementality is the causal difference. It asks: compared to a credible estimate of the counterfactual, how many additional installs did the campaign produce?
Only incrementality answers the last question. Attribution and correlation are useful for other purposes but are not substitutes.
Holdout types
Catchbase supports two test designs. Both pause ad activity for a control period and then reactivate for a test period. They differ in how broad the pause is.
- Full Holdout. All campaigns for the selected app in the selected market are paused during the control period. The test period reactivates only the campaigns or keywords under test. This produces a clean organic baseline and is the more precise of the two designs. It costs more in forgone spend because everything stops.
- Partial Holdout. Only the selected campaigns or keywords are paused during the control period. Other campaigns continue to run. The test period reactivates the paused set. The measurement is less precise because baseline ad activity continues in the background, but the opportunity cost is lower.
Choose Full Holdout when you need a cleaner causal estimate and can afford to pause the market. Choose Partial Holdout when you need to keep other campaigns running and can accept a less precise result.
How the measurement works
The analysis method is CausalImpact (Bayesian structural time series). CausalImpact fits a model to the pre-test period, projects a counterfactual forecast across the test period, and compares that forecast to the observed install series. The difference is reported as incremental installs, with a 90% confidence interval that reflects uncertainty in the model.
Two properties matter for interpretation. First, the method estimates a distribution of outcomes, not a single number. The point estimate is a median; the interval describes the range of values the model finds plausible. Second, the estimate depends on the quality of the pre-test fit. Short or noisy pre-test windows weaken the model, which is why Catchbase requires a minimum of 14 days of pre-test data.