ProgramDistill: From Interactive Web Apps to Verifiable Reference-Guided SWE Tasks
Jeonghye Kim, Minseon Kim, Young Jin Kim, Matheus Pereira, Marc-Alexandre Côté, Alessandro Sordoni, Xingdi Yuan, Zhengyan Shi
cs.SE · cs.AI
Quick verdict
Automated pipeline turns 26 web apps into 4,063 replay-verified agent tasks; strong engineering, but thin statistics and no public release yet.
Summary
ProgramDistill is a benchmark for reference-guided software engineering: coding agents must infer unspecified behavior from a working web application and restore it in an incomplete one. An automated mine-craft-patch pipeline, driven entirely by GPT-5.6 Sol agents, mines replayable browser-interaction traces organized into prerequisite lineages, masks the implementing code, and validates each mask counterfactually (build passes, prerequisites pass, target fails). It yields 1,975 replay-verified behaviors across 26 applications and 4,063 tasks (2,862 atomic, 1,201 cumulative) with no human input. Evaluating nine frontier agents on a 300-task stratified suite, GPT-6 Astra leads with 84.3% mean binary score (versus 68.7% for Claude Opus 5), and success falls from 100%/96% at restoration depth 1 to 64.0%/32% at depth 8. Appendix H.3 shows this decline persists within matched context-length bands. In full-application reconstruction from a scaffold, the best cumulative-workflow score is 49.2%. Trajectory analysis links failures to shrinking per-target observation effort and rising retained stub lines. Main limitations: no variance estimates or significance tests, depth-8 cells as small as 25 tasks, a single construction model, text-only observation, and the benchmark itself not yet publicly released.
Review complete
Assessment at a glance
5
risk flags identified
● high 0
● medium 3
● low 2
5 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- The fully automated mine-craft-patch pipeline converts 26 web applications into 4,063 replay-verified tasks from 1,975 verified behaviors with no human intervention.
- Restoration depth along prerequisite lineages is a controlled difficulty axis: GPT-6 Astra and Claude Opus 5 fall from 100% and 96% at depth 1 to 64.0% and 32% at depth 8.
- The depth-related performance decline persists within matched context-length bands and thus is not explained by longer contexts (Appendix H.3).
- Full-application reconstruction remains far from solved: the best agent recovers only 49.2% of cumulative workflows (58.98% of atomic behaviors).
- Agents devote disproportionately less observation and editing effort per repair target as depth grows, identifying effort allocation as a capability dimension distinct from implementation skill.
Strengths
Counterfactual mask validation gives tasks real F2P/P2P semantics
Section 2.3 admits a mask only if Build(A[m])=1, prerequisite traces still replay (V(A[m],L_{d-1})=1) and the target trace fails; a mask-depth critic rejects shallow masks (feature flags, guards, call-site disabling). This is a far stronger admission criterion than most synthetic SWE benchmarks.
Restoration depth is a genuinely controlled difficulty axis with a confound check
Cumulative tasks compose individually validated masks along prerequisite lineages (1,201 cumulative tasks; 572 LLM-merged), and Appendix H.3 shows the depth-related decline persists within matched context bands (Table 12: -58.4/-54.5/-31.5 points from depth 1 to 8 at 40-80k/80-120k/120-200k prompt sizes), ruling out context length as the sole explanation.
Consistent model ordering across two very different task regimes
Genuinely diagnostic trajectory analysis
Transparent shortcut instrumentation and isolation design
Weaknesses
No statistical treatment of headline results
Benchmark not released; results unverifiable
Single construction model end-to-end
Grading oracles are LLM-curated
Unexplained model anomaly and text-only observations
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
ProgramDistill (arXiv:2609.18805) is a benchmark for reference-guided software engineering: a coding agent receives a working reference web application (source hidden) and an incomplete current application, must infer behavior by interaction, and restore it via code edits verified by replaying recorded browser traces. Two settings are evaluated: partial-application reconstruction (masked repairs on the 300-task ProgramDistill-300 suite) and full-application reconstruction from a minimal scaffold across 12 apps. Central claims: (1) the automated mine-craft-patch pipeline yields 1,975 replay-verified behaviors and 4,063 tasks across 26 applications with no human input; (2) restoration depth along prerequisite lineages is a controlled difficulty axis, with the top agents falling from 100%/96% at depth 1 to 64.0%/32% at depth 8; (3) the decline is not a context-length artifact (Appendix H.3); (4) full reconstruction remains largely unsolved (best cumulative score 49.15%). The paper positions itself against issue-specified SWE-bench-style evaluation and ProgramBench/MirrorCode's whole-program reconstruction.
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 5 strengths & 5 weaknesses, analyzed in detail
- 5 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What is the ProgramDistill benchmark?
ProgramDistill is a benchmark of 4,063 software-engineering tasks generated automatically from 26 interactive web applications. Each task gives a coding agent a working reference app (with source hidden) and an incomplete app, and asks it to infer the missing behavior by interacting with the reference and restore it via code changes. Every task ships with a replayable browser-interaction trace that deterministically checks whether the repaired app behaves correctly.
How do coding agents perform on ProgramDistill?
On the 300-task evaluation suite, GPT-6 Astra leads with an 84.3% mean binary score, Claude Opus 5 scores 68.7%, and GPT-5.6 Sol 60.7%. Success falls sharply with task depth: Astra drops from 100% on depth-1 repairs to 64.0% at depth 8, and Opus 5 from 96% to 32%. In full-application reconstruction from a blank scaffold, the best agent passes only 49.2% of multi-step workflows.
Is ProgramDistill publicly available?
Not at publication time. The paper states the authors are 'working on a public release' of the benchmark, but no code or data link is provided, so its results cannot currently be independently reproduced.
How is ProgramDistill different from SWE-bench?
SWE-bench tasks specify desired behavior through real GitHub issues and tests. ProgramDistill withholds the specification: the agent must discover intended behavior by probing a working reference application, then implement it. Tasks are synthetic — created by automatically masking working code — but each is validated counterfactually so that solving requires restoring real logic and passing replay-based behavioral checks.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- The extracted text omits part of Section 2.4's scoring definitions (chain-score formulas partially truncated) and cuts the Appendix A corpus table mid-row; per-app totals for the last OSWorld rows and the open-source app list were not fully visible.
- Figures are available only as captions and embedded label text; exact curve shapes (e.g., Figure 7b) and the lineage-forest visualizations could not be verified.
- All evaluated models are 2026 commercial systems cited via system cards; their capability claims could not be cross-checked.
- Single-run evaluation is inferred from the complete absence of variance reporting, not from an explicit statement in the text.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.