BAFF: Bid-Aware Filter Family for Mitigating Training Data Interference in RTB A/B Tests
Jeonglyul Oh, Ikkyu Choi, Inseop Youn, Youngjae Kim
cs.LG
Quick verdict
BAFF (k,l) filters curb shared-log bias in RTB A/B tests; promising simulation and DSP evidence, but stats are thin and the bid axis lacks offline support.
Summary
The paper tackles training-data interference in A/B tests for real-time bidding (RTB): when control and treatment models train on a shared serving log, each model's data includes impressions selected and won by its counterpart, biasing the comparison. The authors formalize BAFF, a family of (k,l)-parameterized hard filters that drop counterpart logs on which the two models disagree enough—by ad rank or bid price—interpolating between log-sharing (all data) and log-splitting (none). A ridge-regression analysis bounds the filtered estimator's parameter deviation via three factors, motivating filters that concentrate systematic disagreement in the removed set. Validation combines a synthetic RTB simulation with known ground truth (a 3×3 grid sweep; the strict ad filter k=ε reaches KL 0.51×10⁻³ versus 0.80 for log-splitting and 6.61 for log-sharing) and a live DSP deployment on one advertiser–SSP pair using a three-stage protocol; filter variants stayed within 0.44–5.37 of the reference CPC versus 8.84–9.85 for baselines. Limitations are substantial: the bid axis is effectively untested in the simulation, the online study covers a single advertiser–SSP over three days with very wide confidence intervals (e.g., CPC CI [10.78, 131.49] for the (0.5,0.5) filter), and no code is released.
Review complete
Assessment at a glance
6
risk flags identified
● high 0
● medium 3
● low 3
5 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- Training-data interference in RTB A/B tests enters through two observable channels — ad-ranking and bid-pricing disagreement — and hard filters on these channels can mitigate it without sacrificing all shared data.
- The (k,l) BAFF family interpolates between log-sharing and log-splitting and contains operating points closer to an interference-free reference than either extreme.
- In a 3×3 simulation sweep, the strict ad filter (k=ε) achieves KL 0.51×10⁻³ to the reference policy versus 0.80 for log-splitting and 6.61 for log-sharing (10 seeds).
- In a live DSP deployment on one advertiser–SSP pair, F_1,ε and F_0.5,0.5 preserve reference CPC and CTR more closely (TE 0.44–5.37) than Naive or Log-Split (TE 8.84–9.85).
- The best (k,l) operating point is setting-dependent, so the structured search space itself is the practical deliverable.
Strengths
Well-posed two-channel decomposition
Section 3.1 reduces interference to observable ad-rank (d_ad) and bid-price (d_bp) disagreement scores, both restorable offline by counterfactual rescoring of every logged request — a clean, deployable formalization of the SUTVA violation in RTB.
Sound reference-preservation epistemics
Section 5.3 evaluates strategies by TE against an interference-free reference and explicitly warns that higher CTR or lower CPC is deviation, not gain — the correct framing, rare in industry-originated papers.
Ground-truth simulation plus live production validation
Honest scoping of theory and limitations
Directionally consistent evidence across two settings
Weaknesses
Bid axis has no offline evidence
Online statistics cannot separate the proposed filters
Theory-practice bridge never measured
Simulation wins partly by construction, at toy scale
Simulation's dominant corner not deployed online
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
The paper targets a real production problem: in RTB A/B tests on a DSP, control model A and treatment model B are typically trained on a pooled serving log, so B trains on impressions A selected and won. The authors decompose this symbiosis bias (Brennan et al. [5]) into two observable channels — ad-ranking disagreement (d_ad) and bid-pricing disagreement (d_bp) — and propose BAFF (Section 3.1), a family of (k,l) hard filters that retains an A-log only if d_ad < k and d_bp < l, interpolating between log-splitting (k=0 or l=0) and naive log-sharing (1,1). Three claims: (1) the two-channel decomposition is the right observable proxy for RTB interference; (2) a three-stage online protocol (Section 5.2) measures a strategy's deviation from an interference-free reference; (3) intermediate (k,l) points beat both corner baselines in simulation and live deployment. All three are broadly supported, with material caveats.
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 5 strengths & 5 weaknesses, analyzed in detail
- 6 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What is BAFF in RTB A/B testing?
BAFF (Bid-Aware Filter Family) is a family of hard data filters for A/B tests in real-time bidding. When a treatment model trains on a shared serving log, BAFF drops counterpart-model logs where the two models disagree too much, measured by two tolerances: ad-rank disagreement (k) and bid-price disagreement (l). It offers a tunable middle ground between keeping all shared data (log-sharing) and keeping none (log-splitting).
Why is sharing training data a problem in A/B tests?
If control and treatment models train on the same pooled serving log, each model's dataset contains impressions selected and won by the other model. This violates SUTVA (called symbiosis bias in the literature) and can distort which model wins the A/B test, because each model partly learns from a traffic distribution it would never have generated under its own serving.
Did the BAFF filters improve business metrics like CPC or CTR?
No, and the authors are explicit that improvement is not the goal. Filters are evaluated by how closely they preserve an interference-free reference model's CPC and CTR. In their live DSP test, filters stayed within 0.44–5.37 of the reference CPC while both baselines deviated by 8.84–9.85; the baselines' seemingly better (lower) CPC was itself a bias artifact, not a gain.
Can I reproduce the paper's experiments?
Partially. The synthetic simulation (Section 4, Appendix B) is described in enough detail to reimplement — 100-dimensional context, 30 ads, MLP CTR models, a (k,l) grid over 10 seeds — though no code is released and a few parameters (ad-bias distribution, market competitiveness γ) are unstated. The live DSP experiment on one advertiser–SSP pair cannot be reproduced externally.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Figure 1 appears only as caption text in the extraction (the diagram itself is an image); all quantitative claims in this review were verified against the tables and body text, which extracted cleanly.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.