When Audit Quality Fails to Predict Downstream Utility: A Counterfactual Study of Synthetic-Data Selectors for Low-Resource African NLP
Son Ha Xuan, Phat T. Tran-Truong, Xuan-Bach Le
cs.CL
Quick verdict
Synthetic-data audit rankings decouple from downstream F1 in 8 African-language cells; a transparent, credible case study, though underpowered at a single seed.
Summary
The paper tests whether quality-aware synthetic-data selection signals predict downstream gains in low-resource African NLP. In a matched-budget replay over eight task-language cells (MasakhaNEWS and AfriSenti in Amharic, Hausa, Swahili, Yoruba), five selectors — naive, AlpaGasus-style, DEITA-style, and two CoSDA variants — each retain 64 synthetic examples from one shared, trace-audited candidate pool generated by Qwen2.5-14B; XLM-RoBERTa base is then fine-tuned on 128 rows per cell. Audit channels cover judged label correctness (LC), quality, counterfactual consistency, leakage, shortcut risk, diversity, and a hard-reject decision. The central result is rank disagreement: CoSDA-V2 tops the audit channels (LC 0.904 vs. 0.767 naive; lowest shortcut score) yet AlpaGasus-style selection leads downstream Macro-F1 (0.202 vs. 0.163), with mean within-cell Spearman between LC and F1 of only 0.04. Robustness checks (bootstrap over cells, leave-one-cell-out, non-degenerate subset, non-judge channels) preserve the disagreement. Trace-based analyses attribute the gap to over-pruning, a non-diversity advantage for AlpaGasus, and judge-score saturation. Key limitations: a single seed, generator, judge, and backbone; all bootstrap confidence intervals contain zero; three of eight cells are near-degenerate; and baselines are reimplementations judged by a non-original LLM.
Review complete
Assessment at a glance
6
risk flags identified
● high 1
● medium 3
● low 2
5 strengths · 5 weaknesses · 5 author questions
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- Within each of the eight task-language cells, the Spearman correlation between judged label correctness and downstream Macro-F1 across five selectors averages ρ=0.04 (median 0.00), so audit rank and downstream rank diverge at the cell level, not only in aggregate.
- CoSDA-V2 produces the cleanest audited pool on three channels simultaneously — LC 0.904 vs. 0.767 for naive (+17.9% relative), lowest shortcut score, hard-reject 0.162 vs. 0.486 — yet trails AlpaGasus-style selection downstream (Macro-F1 0.163 vs. 0.202).
- The audit-downstream disagreement is robust to cell-level bootstrap resampling, leave-one-cell-out analysis, restriction to five non-degenerate cells, and the choice of judge vs. non-judge audit channels.
- Three trace-grounded mechanisms explain the gap: audit-strict pruning removes training-relevant support (H1), AlpaGasus's advantage is quality-correlated topical signal rather than diversity (H2, since its D=0.590 is lowest), and judge scores saturate at LC≥0.77 losing rank resolution (H3).
- Methodologically, synthetic-data evaluation should report audit and downstream metrics on the same retained sets, treating LLM-judge scores as audit channels rather than downstream utility proxies.
Strengths
Cleanly controlled comparison design
All five selectors read one shared, trace-backed candidate pool per cell with identical class-balance (±10%), per-seed cap (≤3), and budget constraints (§2-3); gold seeds are fixed, so the selection rule is the only varying factor — a genuinely within-pool test rather than a confounded method comparison.
Exceptional traceability infrastructure
A released claim ledger maps every table cell and in-text aggregate to a row of one authoritative replay CSV (Appendix A, Table 6); per-candidate audit records and per-selector retained pools are published with a deterministic seed-13 replay, making every reported number independently recomputable.
Multi-angle robustness battery
Honest scoping and self-undermining statistics
Mechanism analysis grounded in trace numbers
Weaknesses
Rank statistics computed from n=5 points
Downstream differences are not statistically significant
Baselines are reimplementations under a different judge
Evidence concentrated in degenerate cells
Minor internal inconsistencies vs. the ledger premise
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
- All 5 strengths & 5 weaknesses, analyzed in detail
- 6 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What does this paper show about synthetic data selection?
It shows experimentally that rating synthetic training examples highly with an LLM-based audit does not reliably yield better downstream models. In African-language text classification with tiny training sets, the selector whose retained pool scored best on audit channels (CoSDA-V2: highest judged label correctness, lowest shortcut risk) was beaten on Macro-F1 by a simpler quality-ranking approach (AlpaGasus-style), and the average rank correlation between audit scores and downstream performance was near zero.
What is CoSDA?
CoSDA is the authors' counterfactual audit instrument for synthetic training data. It scores each generated example on five channels — utility, leakage, shortcut risk, diversity, and counterfactual consistency — and can hard-reject examples that violate fixed thresholds. The V2 variant additionally re-ranks surviving candidates with a weighted score before selecting the final training set. In this paper CoSDA is used as a measurement tool, not as a winning selector.
Which languages and tasks were used in the study?
News topic classification (MasakhaNEWS) and sentiment classification (AfriSenti) in four African languages: Amharic, Hausa, Swahili, and Yoruba — eight task-language cells in total. Each cell fine-tunes XLM-RoBERTa base on 64 gold examples plus 64 audited synthetic examples, with Macro-F1 as the metric.
What are the main limitations of this study?
It covers a single random seed, one generator (Qwen2.5-14B), one judge (Qwen2.5-32B), and one downstream backbone (XLM-RoBERTa base), over only eight cells of which three are near-degenerate. All bootstrap confidence intervals for selector-vs-naive differences include zero, the compared baselines are reimplementations rather than original code (with a different judge model than the original AlpaGasus), and no human validation of the synthetic text was performed.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Review is based on extracted text only; Figure 1/2 label-value pairings appear scrambled in extraction, so per-cell correlation values are cited only where the running text states them (e.g., Swahili news +0.87, Hausa sentiment +0.90).
- Appendix tables are interleaved with body text in the extraction; a few per-cell numbers could be garbled and should be cross-checked against the released claim ledger.
- The anonymous artifact repository was not accessed (offline review); reproducibility ratings rest on the paper's own description of released artifacts.
- Appendix B mentions 'hard, soft, relaxed' CoSDA variants that are undefined elsewhere in the extracted text; this may be an artifact of extraction or a genuine gap in the manuscript.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.