Once a test finishes collecting data, the results page summarizes what happened in the holdout and what it implies about the channel’s real contribution. Reading it well means separating the statistical question (is the effect real?) from the business question (is the effect worth the spend?), and knowing which parts of the summary answer which.
What it is
A results readout has three layers: a headline verdict that classifies the outcome, a set of key metrics that quantify it, and a confidence interval that bounds it. Together they describe the range of true lift consistent with what the test observed.
The verdict is one of Significant lift, No detectable lift, or Inconclusive. Each is a claim about signal versus noise, not about profitability. A significant result means the exposed and holdout groups differ by more than chance would explain; it does not, on its own, mean the channel clears your efficiency bar.
ScopeApplies to a single channel, in a single test window, at the holdout size the test was designed around. Results do not generalize across channels, geographies, or seasons without re-testing.
How it works
01The exposed and holdout groups accumulate conversions over the life of the test.
02The gap between them is measured as incremental conversions and incremental revenue, then expressed as lift %, iROAS, and iCAC.
03A confidence interval and p-value describe how precisely that gap is known, and the verdict follows from where the interval sits relative to zero.
Key metrics
How it relates to the verdict
The three verdicts correspond to where the confidence interval sits relative to zero.
An interval entirely below zero is a fourth case: the exposed group underperformed. Before treating that as evidence the channel is harmful, check for assignment or delivery issues.
When a result is inconclusive
Inconclusive is information, not failure. The usual causes:
- Insufficient power. The holdout was too small, or the window too short, to detect the effect size you care about.
- Holdout contamination. Users in the holdout were reached anyway, through a different placement, a retargeting audience, or cross-device delivery.
- Overlapping activity. A promotion, seasonal spike, or another test running in the same window absorbed the signal.
- Effect below the detection floor. The channel may work, just at a magnitude smaller than the test was designed to see.
Adjust the design before you extend the duration. A larger holdout usually buys more power than extra weeks.
Why it matters
A result is a range, not a number. The point estimate sits in the middle of a confidence interval, and the honest planning number is the lower bound. Teams that treat the point estimate as precise systematically over-forecast the return on the channels they measure.
One test describes one moment. Incrementality shifts with saturation, creative, and seasonality. A result from six months ago describes six months ago; ongoing decisions need a re-test cadence, not a single verdict frozen in time.
Statistical significance is not a spend decision. A significant lift still has to clear your marginal efficiency target, not your blended average. The results page tells you whether the channel worked; your target tells you whether it worked enough.
Common pitfalls
Reading the point estimate as precise. Plan against the lower bound of the interval.
Comparing lift across differently designed tests. Different holdout sizes, geographies, and windows produce numbers that are not directly comparable.
Stopping early on a promising trend. Peeking and stopping when results look good inflates false positives. Let the test reach its planned end.
Treating one test as permanent truth. Re-test on a cadence to track drift.
In one sentence
A test result is the range of true lift consistent with what you observed; read the lower bound, compare it to your marginal target, and re-test on a cadence. 
