Campaign Setup
Programmatic experimentation
Plan a programmatic experiment around one decision, comparable campaign conditions, reliable outcomes and a clear interpretation rule.
A programmatic experiment compares buying approaches under a planned split, so the result can inform a campaign decision. Define the change you might make, choose one primary outcome and align the other conditions as closely as the platform allows. Unrelated line items in a dashboard cannot isolate why their results differ.
Write the decision before building the test
State the question in operational terms: what result would justify changing the buy? Define the outcome, its event and counting rule, attribution settings and reporting cut-off. Record guardrails such as authorised spend, suitable placements and acceptable delivery.
Choose a difference worth acting on before seeing results. A small apparent improvement may not justify a more expensive or narrower buy. No universal impression or conversion count makes every test decisive; the information needed depends on the outcome and the decision.
Make the arms comparable
Change one planned factor. Align geography, dates, formats, creative, destination, measurement and required exclusions unless one is the factor under test. Record differences that cannot be removed.
Set each arm's budget in proportion to its audience split and match pacing. Check whether shared budgets or competing line items could affect either arm.
Display & Video 360 offers A/B experiments across insertion orders or line items for audiences, bid strategies and targeting tactics. It can compare combinations of targeting, settings and creatives to discover which perform best.
Experiments assign users to mutually exclusive groups. By default, Display & Video 360 uses user-based identification and random diversion to maximise participation. When a third-party ID is not available, it uses user-based identifiers with backup identifiers: the first-party identifier is used, or otherwise a query-level identifier helps divert an impression into an experiment.
Filtering out users without third-party IDs can reduce cross-arm contamination, but it also reduces the number of users participating. Treat that as a design trade-off: reducing one potential source of contamination may leave less information for comparing the arms.
Trade-offs in Filtering Users Without Third-Party IDs
- Pros
- Reduces cross-arm contamination; improves test integrity by excluding users with uncertain identification.
- Cons
- Reduces the number of participating users, potentially lowering statistical power and generalisability.
Measure the buying result
Read the primary outcome alongside each arm's allocated share, actual spend, impressions and outcome count. Check the denominators behind rates.
A low cost per attributed conversion from a few actions may be fragile; an arm that scarcely spends raises a different question from one that spends fully but produces weak outcomes. Review placement mix and guardrails to see what each approach bought.
Keep recent conversions provisional while reporting and customer actions mature.
A comparison between advertising tactics does not by itself estimate the additional business outcomes caused by advertising compared with no advertising; that requires a suitable lift or other causal design.
Read the range and the count
A reported confidence interval describes a range in which the actual difference between variants is estimated to fall. Display & Video 360 lets you specify a 90% or 95% confidence interval. The interval is not a guarantee that the preferred arm will win again; it indicates the uncertainty around the estimated difference.
Interpret the interval alongside the decision threshold you set before the test. If the range includes outcomes that would favour either arm, or spans both worthwhile and negligible effects, the evidence may not support a firm change. A narrow range can still describe an effect too small to justify the decision.
Keep actual results distinct from normalised results, and put the audience share and actual results beside the normalised figures. Actual values are the raw outcomes received by an arm; normalised values scale its results to a 100% audience split.
For example, 170,000 actual conversions at a 34% audience split would be shown as 500,000 normalised conversions. The latter is a scaled comparison, not the number of conversions actually recorded.
Protect the interpretation
Check that both arms are active, eligible and receiving the intended traffic. Record any creative failure, suitability change or other interruption and its timing.
Where a change is necessary, apply it uniformly to both arms. An unequal mid-test change can alter the question being tested.
At the planned review, adopt the variant, retain the baseline, run a bounded follow-up or record an inconclusive result. State the observed difference, uncertainty, delivery limits and plausible competing explanations.
Pre-Test Checklist for Programmatic Experiments
- Are both arms active and eligible?Yes / No
- Is intended traffic reaching both arms?Yes / No
- Have creative failures or suitability changes occurred?Yes / No – if yes, note timing
- Were mid-test changes applied uniformly?Yes / No
Set a reporting window that fits the data
A reporting cut-off should allow time for outcomes and platform data to mature. In Display & Video 360, impression and click data are updated every 3–4 hours, conversion data roughly every 6 hours, and cross-environment conversion data every 24 hours. These update intervals do not mean every result is final as soon as it appears.
Display & Video 360 may update data for up to 31 days as clicks, impressions and conversions are evaluated for validity. Where the decision can wait, note the report date and review later data before treating a small observed difference as settled. If a decision must be made earlier, record that the result reflects the available data at that cut-off.
Key Data Update Intervals in Display & Video 360
- Impressions & ClicksUpdated every 3–4 hours
- ConversionsUpdated roughly every 6 hours
- Cross-Environment ConversionsUpdated every 24 hours
- Data Validation PeriodUp to 31 days
In this guide
- Testing contextual segments against audience segmentsDesign a contextual-versus-audience programmatic test and interpret differences in reach, spend and campaign outcomes.
- Comparing bidding strategies with matched campaign conditionsSet up a bidding-strategy test with aligned targeting, budgets and measurement, then judge outcomes alongside spend and auction delivery.
- Measuring the effect of placement exclusionsTest an optional placement exclusion and measure its whole-buy effect on delivery, spend, outcomes and replacement inventory.
- Deciding when a programmatic test is inconclusiveIdentify an inconclusive programmatic test using setup validity, outcome counts, uncertainty and the decision it was meant to support.
