Holdout Testing
A testing method where a control group is deliberately excluded from a campaign to measure its true incremental impact.
Why It Matters
Holdout tests provide the cleanest measure of whether a campaign is actually driving results or just taking credit for organic behavior.
How It Works
You randomly assign a percentage of your target audience (typically 10-20%) to a holdout group that will not see the campaign. After the test runs, you compare conversion rates between the exposed group and the holdout to calculate incremental lift.
Real-World Example
A holdout test on a retargeting campaign reveals that 70% of attributed conversions would have happened without the ads, prompting a budget reallocation.
Common Mistakes
Making the holdout group too small for statistical significance
Contaminating the holdout by running overlapping campaigns
Related Terms
An experiment that measures the true causal impact of a marketing activity by comparing a test group that sees it against a holdout group that does not.
The process of assigning credit to the marketing touchpoints that contributed to a conversion.
A statistical method that measures the impact of all marketing channels — including offline — on overall business outcomes like revenue.
Holdout Testing FAQs
What percentage should the holdout group be?
Typically 10-20% of the audience — large enough for statistical significance but small enough to minimize lost revenue during the test.
How long should a holdout test run?
At least 2-4 weeks to account for natural conversion cycles and reach statistical significance.
Need help with holdout testing?
Get matched with a vetted specialist in 48 hours.
Ready to Get Started?
Get matched with a vetted specialist in 48 hours. No recruitment fees, no lengthy hiring process, just results.