Designing an incrementality test
Use this page when you are ready to configure a new incrementality test. The test-creation wizard walks through four steps: app and market, selection of campaigns or keywords, duration, and a final review. The choices you make here determine what the test measures and how precise the estimate can be.
Before you start
- Apple Ads integration connected for the organization
- App Store Connect integration connected, required for incrementality
- Admin role
- At least one active campaign in the app and market you plan to test
Configure the test
Open Incrementality Testing from the left sidebar and click Create Test. The flow is a four-step wizard.
Step 1: App, market, test type. Enter a test name that is unique within your organization. Select one app from the connected Apple Ads account, and one target market (country). The country list is filtered to markets where that app has campaigns. Then pick a test type:
- Full Holdout pauses all campaigns for the app in the selected market during the control period and reactivates only the tested set during the test period. Cleaner estimate; costs more in paused spend.
- Partial Holdout pauses only the selected campaigns or keywords and lets everything else continue. Cheaper; the estimate mixes in ambient traffic from the campaigns that kept running.
Also on this step, choose Campaign Level or Keyword Level. Campaign tests can include one or more campaigns from the selected app and market. Keyword tests must use keywords drawn from exactly one campaign. If you select more than one campaign and then switch to Keyword Level, the wizard keeps only the first campaign and clears the rest.
Step 2: Select campaigns or keywords. For a campaign test, pick the campaigns you want to measure. For a keyword test, pick exactly one campaign and then pick its keywords. When you test more than one item together, the result reflects the combined impact of that set, not the impact of each item individually. To measure a single campaign or keyword in isolation, select only that one.
Step 3: Duration. Pick a start date from tomorrow onward. Set the Pre-Test Period Duration and the Test Period Duration. Each period has a minimum of 14 days; the presets offer 14, 20, and 30 days, and a custom value is allowed as long as it meets the 14-day minimum. The 14-day minimums matter because the model needs enough pre-test data to fit a stable baseline and enough test-period data to detect an effect above normal day-to-day variation. Longer periods produce narrower confidence intervals, at the cost of pausing spend for longer.
Step 4: Review and confirm. The final step shows the full configuration plus the computed control and test date ranges. Catchbase derives the start and end of both the control period and the test period from your start date and the two durations, so you do not enter them directly. Do not create new campaigns, update budgets, add keywords, modify creatives, or manually reactivate paused campaigns while the test is running. Any of those changes during the test window will compromise the measurement.
Choices recap
The decisions the wizard captures, in order:
- Test type: Full Holdout or Partial Holdout.
- Test level: Campaign or Keyword. Keyword tests use keywords from exactly one campaign.
- Campaigns: one or more, on campaign-level tests.
- Keywords: one or more, on keyword-level tests.
- Target market: one country code.
- Start date: tomorrow or later.
- Pre-test period duration: 14 days minimum.
- Test period duration: 14 days minimum.
- App: one app from the connected Apple Ads account.