A/B Testing
Master A/B testing for ML: experimental design, statistical significance, causal inference, and ML experimentation.
What is A/B testing for ML systems?
A/B testing is a controlled production experiment. Eligible users, accounts, or requests are randomly assigned to a control experience or a candidate experience, and the team measures whether the candidate changes a predeclared outcome.
It matters because an offline model score cannot predict the whole product effect. A model can rank examples better yet make the application slower, change who receives an action, or harm a high-value user segment. An online experiment measures that complete path under real traffic.
The core invariant is exchangeability through stable assignment: before treatment, the groups should be comparable; after assignment, the planned treatment should be the only systematic difference between them.
Write the experiment contract before sending traffic
An experiment is a decision protocol, not a dashboard. Its contract freezes what will change, who can enter, what evidence matters, and when the result is complete.
Who is assigned
Experimental unit
Choose a unit that can remain in one variant, such as a user, account, device, session, or request. The unit must match how interference can occur.
What changes
Treatment
Version the model, features, prompt, policy, and serving configuration. Avoid bundling unrelated product changes into the same comparison.
What decides
Outcome contract
Name one primary metric, minimum worthwhile effect, hard guardrails, and material segments before observing results.
When it ends
Stopping rule
Predeclare sample size or a valid sequential rule, minimum runtime, and the actions for win, inconclusive evidence, or harm.
Match the unit to the interference boundary
- Assign by user when repeated visits should receive a consistent experience.
- Assign by account or organization when teammates influence one another.
- Assign by geography or time window when traffic, pricing, or marketplace effects spill across users.
- Assign by request only when repeated exposure cannot change later behavior and cross-request interference is negligible.
Choosing a convenient unit while the treatment acts at a broader boundary creates contamination: control units are partly exposed to the candidate, so the measured difference no longer represents either experience.
Move from a product question to a rollout decision
1 Contract
Frame the decision
Define eligibility, unit, treatment, primary metric, minimum detectable effect, guardrails, segments, and decision owner.
2 Instrument
Validate assignment
Run an A/A check, verify deterministic bucketing, record exposure, and investigate sample-ratio mismatch before reading outcomes.
3 Observe
Run to the plan
Collect the required independent units across a representative time window. Monitor safety without repeatedly changing the success rule.
4 Act
Decide and preserve
Combine effect size, uncertainty, guardrails, segments, and operational cost; then ship gradually, iterate, or roll back with an evidence record.
The sequence matters. Analysis cannot repair assignment that changed across sessions, missing exposure logs, or a hypothesis rewritten after the team saw the result.
Size the experiment around the smallest useful effect
Statistical power is the probability that the planned test detects the minimum effect when that effect is real. Power depends on the baseline rate, effect size, allocation, significance threshold, and number of independent units.
- A smaller minimum detectable effect needs more observations because the signal is harder to distinguish from noise.
- Higher power reduces missed improvements but increases the evidence budget.
- A 50/50 split usually estimates a two-arm difference most efficiently.
- A small treatment allocation limits risk but makes the treatment arm the evidence bottleneck.
Plan the evidence budget
Change the outcome, minimum effect, power, and treatment allocation. The lab recomputes the required units, runtime, and allocation penalty together.
Plan power, allocation, and runtime together
Loading the experiment evidence model...
Reducing the target effect by half usually requires roughly four times as much evidence. If the resulting duration is unrealistic, change the product decision, traffic boundary, or measurement sensitivity before launching.
Keep assignment and exposure reproducible
Randomization creates comparable groups only when assignment is deterministic at the chosen unit and outcomes are joined to actual exposure.
One traceable experiment path
Assignment establishes intent; the exposure event proves that the unit actually encountered the treatment.
Frozen rule
Eligibility
Include only units that satisfy the predeclared population and mutual-exclusion rules.
Deterministic hash
Stable bucket
Hash the experiment ID, assignment unit, and salt into a repeatable control or treatment bucket.
Observed treatment
Exposure log
Record experiment, variant, unit, timestamp, and treatment version when the experience is delivered.
Causal window
Outcome join
Attach eligible outcomes with a predefined attribution window and deduplication rule.
Auditable result
Decision record
Preserve estimates, intervals, segment results, guardrails, exclusions, and the rollout action.
Detect broken experiments before interpreting lift
- Sample-ratio mismatch: observed group counts materially differ from planned allocation.
- Cross-variant exposure: one unit receives both experiences during the measurement window.
- Missing-not-at-random outcomes: outcome capture differs by variant or depends on treatment success.
- Instrumentation drift: metric definitions, event schemas, or attribution windows change mid-test.
Read effect size and uncertainty together
A result is not just positive or negative. The point estimate describes the observed difference, while a confidence interval describes effects still compatible with the data under the analysis assumptions.
Delta
Effect estimate
Candidate outcome minus control outcome in product units
95%
Confidence interval
A conventional uncertainty range, not a probability that the fixed effect lies inside
MDE
Practical threshold
The smallest effect worth the cost and risk of shipping
Guardrail
Independent limit
A safety, reliability, latency, fairness, or cost boundary that lift cannot cancel
Separate four questions
- Is the experiment valid? Assignment, exposure, eligibility, and metric collection passed their checks.
- Is the estimate precise enough? The interval rules out effects that would change the decision.
- Is the effect worthwhile? The plausible gain clears the predeclared practical threshold.
- Is the change safe to expand? Guardrails and material segments remain inside their limits.
Do not turn every secondary metric into another opportunity to declare success. Label metrics as primary, guardrail, diagnostic, or exploratory, and correct the decision rule when several confirmatory hypotheses are tested.
Let guardrails and segments constrain rollout
An aggregate lift can coexist with a serious regression. A large population may dominate the average while a smaller but important segment is harmed, and a better product outcome can arrive with unacceptable latency or reliability cost.
Interpret conflicting evidence
Choose an evidence policy and requested rollout, then inspect how hidden segment harm and guardrail breaches change the maximum safe exposure.
Decide how far conflicting evidence may travel
Loading aggregate, segment, and guardrail evidence...
Operate experiments as production systems
Before launch
- Freeze eligibility, assignment unit, hypotheses, metric queries, segment definitions, and stopping rules.
- Estimate sample size and runtime using expected eligible traffic, not total site traffic.
- Verify mutual exclusion with overlapping experiments and define interaction risks.
- Test exposure and outcome joins with an A/A experiment or replayed events.
During the run
- Monitor sample-ratio mismatch, exposure loss, guardrail breaches, and data freshness.
- Keep treatment versions stable or record every version transition explicitly.
- Avoid unplanned peeking; use a designed sequential method when continuous decisions are required.
- Cover the minimum runtime needed for weekday, seasonality, novelty, and delayed-outcome effects.
After the decision
- Roll out in stages with the experiment identifiers and guardrails still attached.
- Preserve the analysis dataset, query versions, exclusions, estimates, and decision owner.
- Track whether the observed lift persists after novelty and exposure broaden.
- Convert harmful slices and operational failures into regression tests for later candidates.
Know when a standard A/B test is the wrong design
Units affect one another
Network interference
Use cluster, marketplace, or switchback designs when one user's treatment changes another user's inventory, feed, price, or behavior.
Averages are insufficient
Rare high-cost harm
Use pre-release safety evaluation, constrained canaries, expert review, and hard abort rules when a rare failure is unacceptable.
Feedback arrives late
Long delayed outcomes
Use validated leading indicators carefully, preserve long-term holdouts, or choose a design that supports delayed causal measurement.
Review Model Evaluation for offline evidence and promotion gates. Use Prometheus for operational guardrails and MLflow for versioned experiment artifacts; neither replaces assignment, exposure, or causal analysis.