Data Attribution via Sketched Metadifferentiation
Yuxi Chen, Hamza Golubovic, Han Tong, Arian Maleki, Andrew Ilyas
stat.ML · cs.LG
Quick verdict
Casts budgeted attribution as influence-matrix sketching; MAGE/SPELL beat baselines on two modest benchmarks, but core assumptions and scale remain unverified.
Summary
The paper reframes budgeted data attribution as estimating the k×n influence matrix Y of metagradient rows from only B<k replays, exploiting that reverse-mode differentiation through training (REPLAY) with a combined query gradient Vz returns z^T Y in one pass. The authors formalize two objectives — Frobenius reconstruction and the linear datamodeling score (LDS) — and argue they are optimized by different measurements: reconstruction weights queries by squared row norm, attribution by equal weight (a row-normalized surrogate). MAGE probes leading eigenvectors of the query Gram matrix V^T V, approximating the truncated SVD of Y; SPELL targets the row-normalized matrix with warm-up, norm floor, and explicit deflation, each iteration costing one replay plus one forward-mode pass. On ResNet-9/CIFAR-10 (10 seeds, k up to 1000) and a 92M GPT-2-style model fine-tuned on TinyStories (4 seeds, k up to 600), with 300 retrained models per LDS estimate at 1% and 5% removal, MAGE nearly matches the oracle rank-B SVD (e.g., 0.49 vs 0.45 relative Frobenius error at B=50, k=500 on CIFAR-10) and SPELL beats all computable baselines on LDS once B≥0.2k (0.34 vs 0.25 for random probes at B=50, k=500). Appendix C shows row norms span ~8.6 orders of magnitude, explaining the trade-off. Limitations: heuristic LDS surrogate derivations, unverified assumption that the parameter Jacobian preserves query-gradient inner products, only two small-scale settings, and no released code.
Review complete
Assessment at a glance
4
risk flags identified
● high 0
● medium 2
● low 2
5 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- A single metagradient replay fed with any combined query gradient Vz measures the arbitrary row combination z^T Y, so B<k replays suffice to estimate the full k×n influence matrix (Section 3.1).
- Reconstruction and attribution are optimized by different measurements: squared row-norm weighting versus equal weighting of a row-normalized surrogate (Section 3.3).
- MAGE attains reconstruction error nearly matching the oracle rank-B SVD on both benchmarks (e.g., 0.4872 vs 0.4483 at B=50, k=500, CIFAR-10, Table 1).
- SPELL outperforms every computable baseline on the linear datamodeling score once B≥0.2k in both settings (Table 1, Table 2).
- Influence-matrix row norms span ~8.6 (CIFAR-10) and ~7.4 (TinyStories) orders of magnitude, and query-gradient norms predict their ranking with mean Spearman 0.999/0.996 (Appendix C).
Strengths
Clean structural insight with real leverage
Section 3.1 shows one REPLAY fed with a combined query gradient Vz returns z^T Y, converting budgeted multi-query attribution into a matrix sketching problem; the paired forward-mode pass for Y u (Section 3.2) reuses known machinery with proper credit to Gunn (2026).
Explicit two-objective decomposition, empirically grounded
Section 3.3 derives that Frobenius reconstruction weights queries by squared row norm while the LDS surrogate weights them equally; Appendix C validates this with row norms spanning 8.6 (CIFAR-10) and 7.4 (TinyStories) orders of magnitude and a quartile analysis (Table 6) showing MAGE's near-zero top-quartile error vs SPELL's flat profile.
Oracle-relative evaluation quantifies headroom
Disciplined experimental protocol
Actionable practitioner diagnostic
Weaknesses
LDS surrogate theory is an informal chain of approximations
MAGE's key assumption is never measured
Narrow empirical scope
SPELL carries tuned hyperparameters with thin sensitivity analysis
Absolute attribution fidelity at low budget is modest
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
The paper addresses multi-query data attribution: predicting how each of k test queries responds to removing any training example without retraining. Building on MAGIC (Ilyas & Engstrom, 2025), which computes one exact influence row per query via one reverse-mode replay each (cost k replays), the authors recast the problem as estimating the k×n influence matrix Y from B<k measurements. The structural insight (Section 3.1) is that a replay fed with a combined query gradient Vz returns z^T Y — an arbitrary row combination — turning budgeted attribution into a sketching problem. Two objectives are formalized (Section 2.3): Frobenius reconstruction and the linear datamodeling score (LDS). Section 3.3 shows reconstruction is optimized by capturing energy of Y (queries weighted by squared row norm) while LDS is *approximately* optimized by the row-normalized matrix (equal weighting). MAGE probes leading eigenvectors of the query Gram V^T V; SPELL adapts probes for the normalized objective. Claims are properly scoped to two benchmarks: ResNet-9/CIFAR-10 (k≤1000) and a 92M GPT-2-style model fine-tuned on TinyStories (k≤600).
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 5 strengths & 5 weaknesses, analyzed in detail
- 4 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What are MAGE and SPELL for data attribution?
MAGE (Metagradient Approximation by Gram Eigendirections) and SPELL (SPherical Estimation for the Linear datamodeling score at Low budgets) are two algorithms from this paper that estimate the influence matrix linking training examples to many test predictions from a small number of metagradient 'replays'. MAGE is tuned for accurately reconstructing the whole matrix; SPELL is tuned for attribution scores (the linear datamodeling score). They use different probe directions because the two goals weight queries differently.
How much cheaper is this than MAGIC?
MAGIC needs one full reverse-mode pass through training per test query (budget equals number of queries k). MAGE/SPELL need only B<k replays: in the paper's experiments, B=50 replays for k=500 queries on CIFAR-10 already yield useful reconstruction and attribution — roughly a 10× reduction in replay count — with quality rising further at B=100.
What models and datasets were used in the experiments?
Two settings: a ResNet-9 trained on CIFAR-10 (50K training examples, ~89.88% mean test accuracy over 10 seeds, up to 1000 test queries), and a 92M-parameter GPT-2-style decoder fine-tuned on TinyStories (2048 fine-tuning chunks, 4 seeds, up to 600 queries). The linear datamodeling score was measured by retraining the model on 300 random subsets at 1% and 5% data removal.
Is code available for this paper?
The paper text does not mention a public code release. Training hyperparameters are fully tabulated in the appendices and the datasets are public, but a reproducer would need to implement the REPLAY reverse-mode and forward-mode differentiation machinery themselves, which is a substantial engineering effort.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Figures 1-7 appear only as captions in the extracted text; plotted curves, axes, and degradation trends cannot be independently verified.
- Some numeric cells in Appendix C Table 6 look garbled by extraction (e.g., '.92', '0.69.0.88'); quartile-level numbers were quoted cautiously.
- This is a v2 arXiv preprint; no peer-review record or code artifact is referenced in the text.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.