Skill-based Agentic Evaluation for Real-time Data Science Tasks
Aniruddha Tamhane, Raghavendra Addanki, Ayushi Aggarwal, Aditya Bansal, Rui Wang, Rui Wang, Charles Menguy, Swati Jain
cs.AI · cs.LG · cs.MA
Quick verdict
Executable ground truth + factoid scoring for live-data agent eval is promising, but the 53-case, single-skill, self-judged study with perfect recall is thin.
Summary
This Adobe preprint presents an evaluation framework for data-science agents that operate on live, continuously updated data. It combines two ideas: ground-truth-as-code, where each expected answer is an executable Python function re-run against the live data system at evaluation time so the reference drifts with the data; and factoid-level scoring, where an LLM judge decomposes the agent response and the computed ground truth into atomic claims and derives precision, recall, and accuracy from their one-to-one matches, independent of response format, plus rubric-guided Coverage/Tone/Conciseness dimensions on a 0-9 scale. The framework is implemented as a Claude Code harness skill. Validation uses 55 single-turn test cases (53 usable) on two sub-tasks of an in-house production ML skill, backed by a three-vertical synthetic database (~353k records) mirroring production schemas and including decoy datasets. Against three expert annotators (Gwet's AC2 0.883-0.931), the method reaches MCC 0.427 versus 0.331 for a natural-language ground truth and -0.379 for a no-ground-truth self-directed baseline, cutting token cost per case by 16% (~24.6k vs ~29.2k), with case-level recall a perfect 1.000 at precision 0.568. Limitations: judge and skill share the same model (self-preference risk argued but untested), correctness rests on expert-authored ground-truth code, and the appendix worked example retrieves stored reference specs rather than recomputing from live data; no significance testing and no code release.
Review complete
Assessment at a glance
5
risk flags identified
● high 1
● medium 3
● low 1
5 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- Encoding expected answers as executable Python functions recomputed against live data at evaluation time makes agent-evaluation references resistant to data drift (§3.2).
- Factoid-level one-to-one matching of response claims against computed ground truth yields format-agnostic, interpretable precision/recall/accuracy (§3.3-3.5).
- On 53 comparable cases, ground-truth-as-code improves judge-human agreement (MCC 0.427 vs 0.331 for NL ground truth; -0.379 for a no-ground-truth baseline) while cutting per-case token cost ~16% (Table 3).
- The approach generalizes to any agent whose expected output can be expressed as an executable data computation, on any capable LLM and harness (§1, §5).
Strengths
Targets a documented open problem with a sound lineage
The lack of time-varying ground truth is cited as a primary open challenge (Chen et al. 2026a, §1-2); ground-truth-as-code transfers the execution-accuracy philosophy from text-to-SQL (WikiSQL, Spider, BIRD, Spider 2.0, §2) to open-ended agent responses over live data.
Interpretable, format-agnostic diagnostics
Scorecards log hallucinated (Fr\Frg) and omitted (Fg\Frg) factoids verbatim (§3.5); the worked example (Appendix A, Tables 4-5) attributes failures concretely: wrong dataset bundle, a utilization-based target instead of billing.paymentFailed, and unflagged lastPaymentFailedAt leakage.
Measured annotation reliability
Anti-shortcut benchmark design
Candid limitations section
Weaknesses
Perfect case-level recall with unexamined thresholding
No inferential statistics on n=53
Headline mechanism not exercised in the validation
Narrow validation scope
Low reproducibility and loosely controlled baselines
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
This Adobe preprint targets a real gap: evaluating data-science agents whose correct answers depend on live, shifting warehouse data, where frozen benchmark references go stale by construction (§1; Chen et al. 2026a is cited for the open challenge). It contributes (i) ground-truth-as-code (§3.2) — expected answers encoded as typed Python functions re-executed against the live system at evaluation time; (ii) format-agnostic factoid scoring (§3.3-3.5) — an LLM judge decomposes response and computed reference into atomic claims and reports precision, recall, and accuracy over one-to-one binary matches, plus rubric-guided Coverage/Tone/Conciseness on a common 0-9 scale (linear ×9 rescale); and (iii) a human-LLM agreement study (§5) on an internal, production-deployed ML skill. Headline results: MCC 0.427 vs 0.331 for a natural-language reference (a claimed 29% relative gain) and -0.379 for a no-ground-truth self-directed baseline, with ~16% lower token cost per case (Table 3).
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 5 strengths & 5 weaknesses, analyzed in detail
- 5 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What is 'ground-truth-as-code'?
A technique from this paper for evaluating AI agents on live data: instead of storing a fixed reference answer, each test case's expected answer is written as an executable Python function that is re-run against the live data system at evaluation time. The reference therefore stays correct even as the underlying data changes, unlike frozen benchmark answers that go stale.
How much better is ground-truth-as-code than natural-language ground truth?
In the paper's human-agreement study on 53 comparable test cases of an internal ML skill, the LLM judge's agreement with expert annotators (MCC) was 0.427 with executable ground truth versus 0.331 with a natural-language reference (about 29% relative improvement), at roughly 16% lower token cost per case (~24.6k vs ~29.2k). A self-directed baseline without any ground truth correlated negatively with humans (MCC -0.379).
Is the code, benchmark, or data public?
No. The evaluation uses an internal Adobe machine-learning skill and a proprietary synthetic database built to mirror the company's production schemas (353,330 event records across three verticals); the paper does not mention any code or data release.
What are the paper's main limitations?
The same model (Claude Sonnet 4.6) acts as both the skill under test and the judge — a self-preference risk the authors argue away but do not test; correctness depends entirely on expert-written ground-truth code; validation covers only 55 (53 usable) single-turn cases on two sub-tasks of one internal skill; there are no significance tests; and the appendix worked example retrieves a stored reference specification rather than recomputing from live data.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Figure 1 is available only as caption text; the visual pipeline details could not be independently verified.
- Extraction appears complete through Appendix A, but judge prompts and rubric text were not visible, so the qualitative-dimension scoring could not be audited.
- Worked-example factoid counts are approximate (|Fr|≈14, |Fg|≈12, |Frg|≈5), so its per-case metrics carry rounding.
- Item is a preprint ('A PREPRINT' header, arXiv 2609.16487v1, Sep 2026) and appears not yet peer reviewed.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.