The Complexity Kink: A Prompt-Side Structural Complexity Index for Code-Generation Reliability
cs.LG · cs.SE
Quick verdict
Claims prompt-side complexity index exposes failure-dependent measurement in code-gen eval; unusually honest, but its pooled 'kink' is largely compositional.
Summary
This paper tackles failure-dependent measurement in code-generation evaluation: cyclomatic complexity computed on model output is contaminated because failed generations collapse into short, low-complexity programs. The authors build a six-dimension prompt-side rubric (branching, iteration, state, data structures, edge cases, composition; 0-24 composite) scored before generation by four out-of-panel LLM judges (ICC = 0.872) on 5,000 Python prompts from OpenCodeInstruct, then evaluate 21 models with one generation per prompt (105,000 total). Hansen threshold regression on mean-pooled pass rate finds a breakpoint at 13.75 (79.9% vs 87.6%), but the authors' own controls undermine a strong reading: task-type fixed effects shift the breakpoint to 10.75 and cut the gap to 2.1 points; a construction-frame control moves it to 8.50 and reverses the gap; five of 21 model-specific fits move downward. IV diagnostics fail decisively (Sargan J = 411.32), so no causal claim is made. An audit-clean 365-prompt extension matches bins 15-16 but leaves the tail above 16 unresolved (14 prompts). Human agreement is moderate and rater-dependent (r = 0.408-0.921). The paper is a model of transparent reporting, but its central descriptive finding is largely compositional, and the rubric's advantage over a cheap keyword baseline is only partially demonstrated.
Review complete
Assessment at a glance
6
risk flags identified
● high 0
● medium 3
● low 3
5 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- Cyclomatic complexity measured on generated code is failure-dependent (post-treatment): 28.5% of zero-pass generations with computable CC pair a prompt composite > 8 with output CC ≤ 10.
- A four-judge, six-dimension prompt-side composite is a reliable pre-generation index (ICC = 0.872) whose ordering survives paraphrase and Java/C++ re-expression.
- Mean-pooled pass rate shows a breakpoint at 13.75 (79.9% vs 87.6%), but task-type and construction-frame controls shrink or reverse the gap, so it is not a universal failure cutoff.
- The six rubric dimensions fail overidentification restrictions as instruments (Sargan J = 411.32), so the 2SLS fits are diagnostics and no causal claim is made.
- The extreme high-complexity tail above bin 16 remains unresolved, with only 14 audit-clean prompts added.
Strengths
Unusually honest negative-result reporting
The IV failure (Sargan J = 411.32 on 5 df, §6.1), the construction-frame reversal of the headline gap (Table 4: +7.6 → −3.5 points), and the post hoc status of most checks are disclosed prominently rather than buried — a model of calibrated claims.
Scale with panel separation
105,000 generations across 21 diverse models on 5,000 prompts (§4.1); the four rubric judges are excluded from the evaluated panel, limiting scorer-model circularity.
Econometrically serious threshold machinery
Concrete quantification of the measurement problem
Broad measurement-validation battery
Weaknesses
The headline kink does not survive the authors' own controls
Human validation is weak and rater-dependent
Post hoc model exclusion without a rule
Added value over a cheap lexical baseline is only partially demonstrated
Doubly constructed sampling frame
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
The paper targets a measurement problem in code-generation evaluation: cyclomatic complexity measured on generated code is post-treatment — failed generations collapse into short, low-complexity programs, so structurally demanding prompts get sorted into low-complexity bins (the "Reverse Threshold Problem"). The authors quantify this concretely: among 14,776 zero-pass generations with computable Lizard CC, 4,216 (28.5%) pair a prompt composite > 8 with output CC ≤ 10 (§5.4, Fig. 2b). They propose a six-dimension prompt-side rubric (branching, iteration, state, data structures, edge cases, composition; each 0-4, summed to a 0-24 composite), scored before generation by four out-of-panel LLM judges, and characterize how pass rate varies with task structure across 21 models × 5,000 Python prompts = 105,000 generations. The central empirical claim — a nonmonotone "kink" with breakpoint at 13.75 — is presented with unusual care and immediately qualified by the authors' own sensitivity analyses.
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 5 strengths & 5 weaknesses, analyzed in detail
- 6 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What is the 'Reverse Threshold Problem' described in this paper?
When task complexity is measured from the generated code, failed generations (short stubs, syntax errors) receive artificially low complexity scores, so structurally hard prompts get sorted into low-complexity bins. The paper quantifies this: 28.5% of failed generations pair a structurally nontrivial prompt (composite > 8) with output complexity ≤ 10.
Is there a prompt complexity level at which code-generation models become more reliable?
The paper finds a pooled breakpoint at composite 13.75 (pass rates 79.9% below vs 87.6% above) but explicitly shows it is not a universal cutoff: task-type controls shrink the gap to 2.1 points, a construction-frame control reverses it to −3.5 points, and 5 of 21 models move downward.
How reliable is the LLM-scored complexity rubric?
The four-judge ensemble reaches ICC = 0.872, and score ordering is preserved under paraphrase and cross-language re-expression, but agreement with human graders is moderate and varies by rater (Pearson 0.408 to 0.921).
Does the paper claim complexity causally affects code-generation reliability?
No. The six rubric dimensions fail Sargan and Wooldridge overidentification tests as instruments (J = 411.32 and 350.59), so the 2SLS estimates are diagnostics only and the analysis is explicitly observational.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Figures 1-6 could not be visually inspected; assessments rely on in-text descriptions and numeric tables.
- The evaluated model list includes model names beyond verifiable knowledge (e.g., GPT-5.4, Gemini 3.1 Pro Preview); reported results are taken at face value.
- The analysis artifact is referenced but not inspected; reproducibility claims rest on the authors' description.
- The abstract in the provided text is truncated mid-sentence; conclusions drawn from the full body text, which appears complete.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.