Let Training Guide Selection: Online Synthetic Data Filtering via Real-Anchored Utility
Yanran Wu, Sana Lakdawala, Renzo Tassara Miller, Chongyang Bai, Sharath Ciddu, Shivendra Pratap Singh, Kungang Li, Sandeep Pandey, et al.
cs.LG
Quick verdict
FROST filters synthetic data online via real-gradient alignment; gains span vision, text-to-SQL and ads, but results are single-run with no variance tests.
Summary
The paper introduces FROST, an online framework for filtering synthetic training data by estimating each synthetic sample's utility through gradient alignment with an exponentially moving average of real-training-batch gradients, computed on a small parameter block (prediction head or final backbone layer). A batch-level gate calibrates mean utility against a rolling buffer of recent batch scores via a z-score, retaining in-band batches whole; out-of-band batches undergo sample-level IQR (Tukey-fence) filtering. No external verifier or held-out validation set is required. Experiments cover CIFAR-100 with three image generators (FLUX.1, SANA, SD1.4), text-to-SQL fine-tuning of Qwen2.5-Coder-0.5B on Spider/SynSQL, and a production ads re-ranking system. FROST removes roughly 20–30% of synthetic data while improving accuracy over full-pool training (+0.58 to +1.14 pp on images; +0.9 pp execution accuracy on Spider), beating Random, DS3, CovMatch, InfoBatch, OrderDP, GradNorm, and a modified GREATS at matched drop ratios. In the ads system, FROST converts a +0.211% normalized-entropy regression into a −0.096% improvement while dropping 36% of synthetic data, at 5.7–6.8% throughput overhead. Main limitations: all headline comparisons are single-run without variance estimates or significance tests, the industrial experiment is not reproducible (private data, minimal detail), hyperparameter provenance is unclear, and the GREATS baseline was modified to drop its validation reference.
Review complete
Assessment at a glance
4
risk flags identified
● high 1
● medium 3
● low 0
5 strengths · 5 weaknesses · 5 author questions
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- Gradient alignment with a bias-corrected EMA of real-training-batch gradients yields an effective online utility signal for filtering synthetic data, without an external verifier or held-out validation set.
- Conditioning sample-level IQR filtering on batch-level z-score routing outperforms either stage alone on both public benchmarks and the industrial ads task.
- FROST removes 20–30% of synthetic data while improving real-task performance over full-pool training: +0.58 to +1.14pp accuracy on CIFAR-100 across three generators and +0.9pp execution accuracy on Spider dev.
- In a production ads re-ranking system, FROST converts a +0.211% normalized-entropy regression from full synthetic-pool training into a −0.096% improvement, exceeding the 0.02% industrial significance threshold, at 5.7% throughput cost.
Strengths
Two-stage design is empirically justified by ablations
Table 5 / Figures 3 and 5: batch-gate-only drops 80% of synthetic data and loses 5.0pp EX vs full FROST (60.7 vs 65.7); sample-IQR-only drops 32.2% and loses 2.6pp. On the ads system (Fig 5), batch gate only lands at +0.074% and IQR-only at +0.036% vs −0.096% for the combination — the routing idea earns its complexity.
Cross-domain parameter transfer suggests non-fragile defaults
Section 4 preamble: the same band (−0.025, 1) and IQR multipliers (0, 1.5) are used for both CIFAR-100 and text-to-SQL without domain-specific tuning, and results hold across three different generators and two model families (ResNet-18, Qwen2.5-Coder-0.5B).
Honest, informative negative and exploratory results
Theory is scoped to what the score actually guarantees
Above-average implementation transparency and cost accounting
Weaknesses
No variance or significance reporting anywhere
Key baseline is modified, possibly handicapped
Hyperparameter provenance is opaque and partly post-hoc
Industrial experiment is unverifiable
LLM generalizability rests on one small model and one task
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
FROST addresses synthetic-data selection at training time: given mixed real/synthetic mini-batches, decide which synthetic samples to keep. The claimed novelty is twofold — utility estimation via gradient alignment with an EMA-smoothed real-training gradient reference (TracIn-inspired, Section 3.1), and a conditional two-stage filter where a batch-level z-score gate against a rolling buffer (Section 3.2) routes only out-of-band batches to sample-level IQR filtering (Section 3.3). The paper claims model- and task-agnostic operation with no external verifier and no held-out validation set, supported by three settings: CIFAR-100 with three generators, Spider text-to-SQL with Qwen2.5-Coder-0.5B, and a production ads re-ranking system. The headline numbers — +0.58 to +1.14pp image accuracy (Table 1), +0.9pp Spider EX (Table 2), and flipping a +0.211% NE regression into −0.096% (Table 3) — are stated accurately relative to the tables.
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 5 strengths & 5 weaknesses, analyzed in detail
- 4 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What is FROST and what problem does it solve?
FROST (Filtering with Real data for Online Synthetic-data Training) is a framework that filters synthetic training data while the model is being trained on a mix of real and synthetic batches. It scores each synthetic sample by how well its gradient aligns with a smoothed average of real-data gradients, keeps batches whose average score sits in a moderate band, and filters individual samples only in suspicious batches. The goal is to remove the roughly 20–30% of synthetic data that is noisy or mismatched, so training on the remainder beats training on the full synthetic pool.
How much does FROST improve performance?
On CIFAR-100 image classification with three generators (FLUX.1, SANA, SD1.4), FROST improves accuracy by 0.58 to 1.14 percentage points over training on the full synthetic pool while dropping about 25% of synthetic images. On text-to-SQL (Spider 1.0 dev with Qwen2.5-Coder-0.5B), it gains 0.9 points of execution accuracy while dropping 20.4% of the synthetic pool. In a production ads re-ranking system it converts a +0.211% normalized-entropy regression into a −0.096% improvement, above the 0.02% gain considered significant in prior industrial studies.
Does FROST need a validation set or a separate data-quality model?
No. The utility reference is built from the real training batches themselves via an exponential moving average of their gradients, thresholds come from a rolling window of recent batch scores plus within-batch quartiles, and no external verifier scores the synthetic data. This distinguishes it from methods like LESS and GREATS, which use held-out target examples as the selection reference.
Is FROST code or data publicly available?
The paper does not mention a code release. The public experiments rely on available resources (CIFAR-100, the FLUX.1/SANA generators, the Shipard et al. SD1.4 image pool, Spider 1.0, and SynSQL), and the appendix gives detailed hyperparameters and prompt templates. The industrial ads experiment uses proprietary Meta data and cannot be reproduced externally.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Figures 1–5 are extracted as partial text; plots (Fig 2 accuracy curves, Fig 4 band sensitivity) are interpreted from embedded captions and inline numbers only.
- Appendix B.2 appears truncated mid-sentence ('followed by 18 cosine decay'); some implementation details may be missing from the extraction.
- Minor OCR artifacts exist in equations and figure text; all quoted numbers were cross-checked against tables and prose where possible.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.