Spec2COBOLRot: An Agentic-AI Degradation Loop for Realistic COBOL Corpus Generation
Jean-Baptiste Espinasse, Djamel Eddine Khelladi, Mathieu Acher
cs.SE · cs.AI
Quick verdict
Proposes a two-phase agentic pipeline generating degraded COBOL benchmarks; honest about 5/9 behavior preservation and thin, single-domain evaluation.
Summary
The paper addresses the scarcity of realistic COBOL corpora for benchmarking AI-based modernization tools. The authors propose Spec2COBOLRot, a two-phase agentic pipeline: Phase 1 generates a clean, compilable COBOL program from a natural-language specification using GPT-5.4, validated by a GnuCOBOL compile-and-execute repair loop; Phase 2 iteratively injects legacy patterns from a 26-asset library (8 idioms, 15 smells, 3 mismatches) extracted from 14 proprietary production payroll programs, stopping when eight structural metrics (SLOC, NPAR, DIVR, CC, APL, NGO, VUR, DPR) reach target intervals derived via linear regression from the reference corpus. Across three specifications (banking, payroll, insurance) with three runs each, all Phase-1 programs compiled and executed, and degradation moved programs into the target bands on nearly every metric axis for complexity levels L1-L2. However, functional behavior was preserved in only 5 of 9 runs, verified by output diffing on one synthetic dataset. Experts (2-3 practitioners with 20-30 years' experience) judged the programs partially plausible but 'too clean', missing patterns like COPY statements and implicit period termination; at larger target sizes (L3-L4) the model tends to duplicate near-identical paragraphs, eroding realism. The authors candidly discuss these limits and propose a simulated development history as future work. Main limitations: tiny evaluation scale, single-domain reference corpus, proprietary metric tools, no baseline comparison.
Review complete
Assessment at a glance
6
risk flags identified
● high 0
● medium 3
● low 3
4 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- A two-phase pipeline (specification-driven generation plus asset-guided iterative degradation) reliably produces syntactically valid, executable COBOL from natural-language specifications (RQ1).
- Iterative degradation moves generated programs into corpus-derived target complexity bands on nearly every metric axis for levels L1–L2 (RQ2, Figure 3).
- Business behavior preservation is not achieved by construction — only 5/9 degradation runs preserved outputs — motivating a tighter per-iteration verification loop.
- Targeting structural metrics independently of business logic has realism limits: the LLM duplicates near-identical paragraphs at larger targets (RQ3) and experts judge the results 'too clean' (RQ4).
- A simulated development history driven by a multi-decade maintenance plan is proposed as a more realistic future alternative to direct pattern injection.
Strengths
Honest negative reporting
The paper foregrounds its own failures: 5/9 functional preservation (Section 6.2), the experts' 'too clean' verdict with missing COPY/REDEFINES patterns (Section 6.4), and paragraph duplication gaming size targets (RQ3). This candor, unusual in corpus-generation work, raises trust in the reported numbers.
Grounded, reusable realism calibration
Eight metrics (Table 1) and the correlation-based interval derivation (Section 4.3, Table 5) require only structural metrics computable on any production corpus; the SLOC-anchored regression with R² > 0.7 gating, ±1 residual SD widening, and clipping rules is specified precisely enough to transfer.
Generation/degradation separation enables known ground truth
End-to-end executability discipline
Weaknesses
Unsupported L3/L4 claims
Weak functional-equivalence methodology
Circular calibration, no discriminant validation
No baseline comparison
Tiny evaluation and thin expert review
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
This AISM '26 workshop paper (8 pages) proposes Spec2COBOLRot, a two-phase agentic pipeline that manufactures 'realistic' COBOL benchmark programs, where realism is explicitly operationalized as structural fidelity: eight metrics (SLOC, NPAR, DIVR, CC, APL, NGO, VUR, DPR — Table 1) matching ranges observed in 14 proprietary production payroll programs from Sopra Steria (Section 3, Table 2). Phase 1 generates a clean, compilable program from a natural-language specification (GPT-5.4, GnuCOBOL compile/execute repair loop, k = 5); Phase 2 iteratively injects legacy patterns from a 26-asset library (8 idioms, 15 smells, 3 mismatches — Table 4) until metrics enter target intervals derived by SLOC-anchored linear regression (Table 5). Central claims: (1) the pipeline reliably yields valid, executable programs (RQ1); (2) degradation reaches corpus-calibrated complexity targets (RQ2); (3) behavior preservation is not guaranteed (5/9 runs); (4) metric-only targeting has realism limits (RQ3, RQ4). The abstract and conclusion are notably honest about claims 3 and 4.
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 4 strengths & 5 weaknesses, analyzed in detail
- 6 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What is Spec2COBOLRot?
It is an agentic AI pipeline for creating realistic COBOL benchmark programs. It first generates a clean COBOL program from a natural-language business specification (validated by compiling and running it with GnuCOBOL), then iteratively 'degrades' the code by injecting legacy patterns (GOTOs, dead code, over-declared variables) taken from real production programs, until structural complexity metrics fall within ranges observed in an industrial corpus.
Does Spec2COBOLRot preserve the program's behavior?
Not reliably. In the paper's evaluation, only 5 of 9 degradation runs (three business specifications, three runs each) produced identical output files to the clean reference program on the test dataset. The authors recommend adding a per-iteration behavior check, as already used in the generation phase.
Why generate synthetic COBOL instead of using real code?
Real enterprise COBOL is almost always proprietary and cannot be shared, while public COBOL collections (GitHub repositories, algorithmic exercises) lack production characteristics such as copybooks, indexed file handling, and decades of maintenance drift. Synthetic corpora with calibrated complexity aim to fill this benchmarking gap for modernization tools.
Is the code and data available?
A replication package with samples exists on GitHub (jbespi/spec2cobolrot-samples). However, the reference corpus (14 payroll programs) and the structural metric tools come from an industrial partner and are not public, and the pipeline uses non-deterministic GPT-5.4 calls, so exact reproduction is not possible.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Figures 1 and 3 are only partially extractable; per-metric pre/post values in Figure 3 could not be verified against the 'nearly every axis' claim.
- No L3/L4 result tables appear in the extracted text — either absent from the paper or lost in extraction; RQ3 claims could not be checked against data.
- Asset library contents and prompts live in the external replication package and were not auditable from the provided text.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.