CordisBench: Can Language Models Reason About Component Lifecycles in Dynamic Agent Harnesses?
cs.CL · cs.AI
Quick verdict
CordisBench tests LLM reasoning on agent-harness component lifecycles; rigorous executable design, but 3 models and one interference pattern limit scope.
Summary
The authors introduce CordisBench, a 1,200-question benchmark (from 240 generated systems, three replicates) probing whether language models can reason about component lifecycles—dependencies, cleanup effects, and order-sensitive teardown—in dynamic agent harnesses. Two settings are combined: a formal one with finite reference semantics, and a Cordis-native one executed against the Cordis runtime (4.0.0-rc.7). Four task families (localization, schedule prediction, guaranteed and reachable conditions) plus executable reconfiguration and outcome-count diagnostics are evaluated at interaction counts 2 through 32. Three efficiency-oriented models (Gemini 3.7 Flash, GPT-5.6 Luna, DeepSeek V4 Flash) are tested at low reasoning effort. Key results: localization degrades more slowly than final-state prediction and cross-order reasoning; GPT-5.6 Luna's formal reachable-condition Jaccard falls from 91.7% to 14.1% and executed reconfiguration success from 62.5% to 25.0%; DeepSeek adopts a return-all strategy on condition tasks. The finite reference semantics agrees with Cordis execution on all 528 executable questions. Raising reasoning effort helps GPT-5.6 Luna substantially (prediction 31.2% to 85.4% at 16 interactions) at roughly 3,000 reasoning tokens per question. Limitations: a single interference pattern, three models, synthetic systems, and per-cell samples of only a few dozen questions.
Review complete
Assessment at a glance
4
risk flags identified
● high 0
● medium 2
● low 2
5 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- As relevant interactions grow from 2 to 32, localization degrades more slowly than final-state prediction and reasoning across teardown orders (Figure 3).
- For GPT-5.6 Luna at 100% parse rate, formal reachable-condition Jaccard falls from 91.7% to 14.1% and Cordis-native executed reconfiguration success from 62.5% to 25.0% with scale (Section 5.1).
- A finite reference semantics agrees with Cordis execution on every scored observation and action outcome across all 528 executable questions (Section 5.3).
- Additional reasoning effort recovers large gains for GPT-5.6 Luna on the 16-interaction subset (native prediction 31.2% to 85.4%) at ~2,967 reasoning tokens per question (Section 5.2).
- Execution-based scoring exposes non-minimal plans: DeepSeek V4 Flash attains 33/96 targets but only one minimal success (Section 5.4).
Strengths
Execution-verified ground truth
Reconfiguration answers are translated into Cordis dispose(...) calls and executed rather than string-matched (Section 3.4, 5.4), and the finite reference semantics agrees with Cordis execution on every scored observation/action across all 528 executable questions (Section 5.3) — unusually rigorous for an LLM benchmark.
Task-matched metrics with shortcut controls
Jaccard, per-observable accuracy, and executed success replace mechanically stricter whole-answer exact match (Section 4); best degenerate control reaches only 7.3% (Section 3.5), and the 50% return-all reference line exposes DeepSeek's degenerate all-labels strategy from size 8 onward (Section 5.1).
Explicit confound diagnostics
Statistically aware design
Failure-mode decomposition
Weaknesses
Single interference pattern
Effective single-model scaling narrative
Unexplained non-monotonic cells
Effort/cost ablation is narrow
Contamination unaddressed
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
CordisBench — Structured Preliminary Review
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 5 strengths & 5 weaknesses, analyzed in detail
- 4 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What is CordisBench?
A 1,200-question benchmark from Univ. Lille/Inria researchers that tests whether large language models can predict what happens when an AI agent adds, removes, or reconfigures components (plugins and services) in its own runtime harness, where removing one component can force dependent shutdowns and cleanup effects whose final result depends on teardown order. Model answers are checked against the real Cordis runtime, and reconfiguration plans are actually executed.
What did the CordisBench study find?
At low reasoning effort, the three tested models (Gemini 3.7 Flash, GPT-5.6 Luna, DeepSeek V4 Flash) handle small configurations well but become unreliable as more interactions matter, especially when predicting final state or reasoning across teardown orders. GPT-5.6 Luna's formal reachable-condition Jaccard fell from 91.7% to 14.1% and its executed reconfiguration success from 62.5% to 25.0% as size grew from 2 to 32 interactions. More reasoning tokens help but cost nearly 3,000 tokens per question.
Is CordisBench data and code public?
The paper lists GitHub and Hugging Face resources. The benchmark is synthetically generated with deterministic scoring, so the benchmark machinery is reproducible; however, re-running the full evaluation requires paid API access to the three proprietary frontier models tested.
Why does teardown order matter in this benchmark?
Two components can each save the value of a shared setting when they start and restore that saved value when they stop. Each behaves sensibly alone, but stopping them in different orders restores different captured values, so the final state depends on the teardown order. CordisBench scales this interference pattern up to 32 interacting components and asks models to reason about it exactly.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Numeric values along the Figure 3/4 curves are only present as rendered plots in the PDF text; this review relies on in-text numbers and appendix Tables 2-5.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.