Between the Commits: Process, Error, and Claim Reliability in a Wholly AI-Authored Codebase
cs.SE · cs.AI
Quick verdict
First process-level error metrics from a wholly AI-authored codebase (14.3% event errors; 1-in-4-5 responses wrong) - credible, with single-project and self-judging caveats
Summary
The authors instrument the complete development history of Extractor, a 21,000-line Python tool (recovering protobuf schemas from compiled Apple binaries) written entirely by Claude models with no human code or tests: 210 commits, 25 sessions, 678 instructions. They contribute the dataset plus two provenance tools (CFG-block decomposition with git log -L tracing; a session-transcript miner for pytest FAIL-then-PASS episodes) and three taxonomies for instruction intent, commit trigger, and response reliability, validated by dual-LLM annotation (κ 0.706-0.808) and human calibration samples. Development proves mainly proactive (86-90% of commits). 14.3% of 105 auditable code-generation events introduced an error caught pre-commit, but only 3.8% were production-logic defects; most were test bugs or propagation misses. Fact-checking roughly 3,900 response components yields 94.3% accuracy for checkable facts, 91.1% for mechanism explanations, 89.3% for root-cause diagnoses, and 79.4% for design-fix proposals; about one response in four to five contains an inaccuracy. Limitations: one project, one expert user, one model family; LLM judges with limited human calibration; unreported exclusions.
Review complete
Assessment at a glance
0
risk flags identified
● high 0
● medium 0
● low 0
4 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- User instructions to CLI coding agents differ in kind from IDE-chat instructions, with far more comprehension, planning, consultation and validation (Finding 2 of Tang et al. is not supported: Failure Reporting 1.77% vs 24.0%).
- Code development is mainly proactive: 86-90% of commits and 89-90% of CFG code blocks trace to proactive triggers rather than to bugs or test failures.
- 14.3% of auditable AI code-generation events introduced a real error caught by the AI-authored test suite before commit, though only 3.8% were production-logic defects; most were test bugs or propagation misses.
- Response accuracy declines with claim difficulty: 94.3% for checkable facts, 91.1% for mechanism explanations, 89.3% for root-cause diagnoses, 79.4% for design-fix proposals.
- Roughly one in four to five interactive responses contains at least one factual inaccuracy (23.0%/21.6% excluding design-fix components; 29.5% including them).
Strengths
Unprecedented process-level dataset
Commits + all 25 session transcripts + AI process artefacts for a 21k-LOC codebase; the paper shows existing corpora (SWE-chat: median 8 sessions/0% AI commit share; AIDev: PR metadata only, Aug 2025 cutoff) cannot support process-level analysis (§II, Fig. 1).
Pre-commit error visibility is genuinely new
The pytest FAIL→edit→PASS miner (§V) recovers mid-session corrections that commit-based studies (SZZ variants, PR-mining) structurally cannot see; human verification of every defect episode caught two misclassifications, and mutation-testing false positives are explicitly filtered.
Methodological transparency and validation
Honest handling of uncertainty
Weaknesses
Abstract overstates the code-error evidence
Response accuracy denominator choices all favour accuracy
External validity: n=1 everything
Potential judge contamination via session transcripts
Human-baseline comparison is asymmetric
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
The paper instruments the complete development of **Extractor** (`apple_protobuf_extractor`), a 21,000-line Python tool that recovers protobuf schemas from compiled Apple binaries, written entirely by Claude models with no human code or tests (210 commits, 25 sessions, 678 instructions; Table I). Three contributions are claimed: (i) a full-process dataset (commits + session transcripts + AI-generated notes); (ii) two provenance tools; (iii) three taxonomies for instruction intent, commit trigger, and response reliability; plus first-of-kind error measurements: 14.3% of auditable code-generation events erroneous, and roughly 1-in-4-5 interactive responses containing an inaccuracy.
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 4 strengths & 5 weaknesses, analyzed in detail
- 0 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What is the Extractor dataset?
It is the complete development history of a 21,000-line Python tool (apple_protobuf_extractor) written entirely by Claude AI with no human-authored code, tests, or commits. The public GitHub package contains all 210 commits, 25 full AI session transcripts, 1,467 tests, AI-generated process notes, the three taxonomy codebooks, the scoring rubric, raw LLM classification/verdict outputs, and scripts to regenerate every table and figure in the paper.
How often did Claude's generated code contain errors in this study?
14.3% of auditable AI code-generation events (15 of 105) introduced an error that the AI-written test suite caught before any commit. Only 3.8% (4 events) were genuine production-logic defects; 7.6% were bugs in the newly written tests themselves and 2.9% were failures to propagate a correct change to dependent tests.
Is there a human baseline for the 14.3% error rate?
This is believed to be the first measurement of pre-commit errors caught by tests in fully AI-authored code. Commit-based analyses structurally cannot see these, and no comparable human baseline exists because standard defect-density measurement only observes code after it is committed.
Can Claude's responses during coding sessions be trusted?
It depends on the claim type: factual statements were 94.3% accurate, explanations of existing mechanisms 91.1%, root-cause diagnoses 89.3%, and design-fix proposals only 79.4%. Overall, roughly one in four to five interactive responses contained at least one factual inaccuracy; counting design-fix proposals the figure rises to 29.5%.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Paper text appears complete but all figures/tables are analysed from extracted text only; exact bar values in Figs 3, 7, 8, 9 are read from axis labels and may carry extraction noise
- Some taxonomic detail (full codebooks, rubric) lives only in the external dataset package and could not be inspected
- arXiv 2609.29744 is recent (Sep 2026); no independent citations or published commentary on it were available to this review
- Model names in the text (e.g. 'Claude Sonnet 5', 'Codex GPT-5.6 Sol') are taken verbatim from the extracted text
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.