FIRE: Failure-Informed Runtime Engineering for Reliable Language-Model Agents
cs.AI · cs.CL · cs.SE
Quick verdict
Claims runtime policies from observed failures lift agent pass^2 on Terminal-Bench 2.1 across 3 GPT-5.6 tiers; rigorous design, honest stats, small N.
Summary
The paper proposes runtime policies: targeted natural-language instructions and a few action denials that the agent harness injects at trajectory states that preceded observed failures, without changing model weights or the user prompt. Policies pair an eligibility predicate over task descriptions with runtime predicates over harness events, organized into frozen families (service persistence, database-evidence denial, exact replacement; Table 1). On 87 tasks of Terminal-Bench 2.1 (two qemu tasks excluded), two attempts per task, with Codex CLI across three GPT-5.6 tiers, repeated success (pass^2) rises in every tier: Luna 50.6→54.0%, Terra 55.2→60.9%, Sol 64.4→73.6%, while Sol's pass@2 moves only 1.2 points, supporting the claim that policies convert reachable solutions into dependable delivery. A randomized five-arm panel on 14 eligible Terra tasks separates content from timing: real policies 60.7% versus 35.7% for a timing-matched sham (prespecified primary contrast +25.0 points, 95% CI [7.1,46.4], p=0.061), with generic verify/reconsider arms at 39-43%. On the eligible cohort Terra+policy reaches 71.4% versus unassisted Sol's 64.3% at about half the cost ($7.19 vs $14.48). Main limitations: small samples (14 tasks, two attempts), the primary contrast misses p<0.05, policies are derived from the same benchmark's development population, one harness and model family, both authors are Failproof AI co-founders, and full-suite policy overhead raises Sol's cost from $111 to $165.
Review complete
Assessment at a glance
5
risk flags identified
● high 0
● medium 3
● low 2
5 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- Runtime policies derived from observed failures raise repeated success (pass^2) on the 87*-task Terminal-Bench 2.1 suite in all three GPT-5.6 tiers (Luna 50.6→54.0%, Terra 55.2→60.9%, Sol 64.4→73.6%).
- The gain comes from policy content, not interruption timing or generic self-checking: in the randomized five-arm panel, a timing-matched sham and verify/reconsider arms fail to reproduce it (primary contrast +25.0 points, p=0.061).
- Policies chiefly convert reachable solutions into dependable delivery: Sol's pass@2 changes by only 1.2 points while pass^2 rises 9.2 points.
- On the 14 tasks covered by Terra's frozen portfolio, policy-guided Terra (71.4%) reaches the range of unassisted Sol (64.3%) at about half the cost ($7.19 vs $14.48).
Strengths
Causal design that isolates policy content from interruption timing
The five-arm panel (real, fresh baseline, timing-matched sham, always-verify, reconsider) with preregistered primary contrast, hashed task sets and policy sources committed before the first attempt, and silent-task controls cleanly separates what a rule says from the fact of being interrupted (§4, Table 2)
Exemplary statistical honesty
Reports the primary contrast as +25.0 points with CI [7.1, 46.4] but p=0.061, calls it 'a large effect with a wide interval, not a settled one', and refuses to choose between disagreeing interval and test (Appendix B) — rare candor in agent evaluation
Internally consistent metric decomposition
Artifact and lineage transparency
Mechanism and cost evidence, not just end metrics
Weaknesses
Thin statistical power throughout
Same-benchmark policy derivation leaves overfitting unexcluded
Narrow external validity
Vendor conflict of interest
Full-suite overhead sits in tension with the cost headline
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
FIRE studies runtime policies: short natural-language instructions and two denial rules injected by the agent harness at trajectory states that preceded observed failures, with model weights and user prompts untouched. Each policy pairs an eligibility predicate over the task description with runtime predicates over harness events (prompt submit, pre-tool-use, stop), grouped into eight frozen families (Table 1: service persistence, deployment preservation, database evidence, exact replacement x3 rules, LaTeX final log, binary equivalence x2, token accounting, enumeration). Four claims: (1) pass^2 rises in all three GPT-5.6 tiers on the 87*-task Terminal-Bench 2.1 suite (Luna 50.6→54.0%, Terra 55.2→60.9%, Sol 64.4→73.6%); (2) the mechanism converts reachable solutions into dependable delivery (Sol pass@2 +1.2 only); (3) a randomized five-arm panel attributes the gain to policy content, not interruption timing or generic self-checking (real 60.7% vs sham 35.7% on 14 eligible tasks; primary contrast +25.0 points, 95% CI [7.1, 46.4], p=0.061); (4) on that cohort, policy-guided Terra (71.4%, $7.19) reaches unassisted Sol (64.3%, $14.48) at about half cost.
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 5 strengths & 5 weaknesses, analyzed in detail
- 5 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What are runtime policies in this paper?
Natural-language instructions and a few action denials that the agent harness injects at trajectory states which previously led to observed failures, without changing model weights or the user prompt. Policies are built by clustering observed failures into mechanism families (service persistence, database-evidence protection, exact replacement, etc.) and writing predicates that fire only when the risky state recurs.
How much do the policies improve agent reliability?
On the 87-task Terminal-Bench 2.1 suite with two attempts per task, repeated success (pass^2) rose from 50.6% to 54.0% for Luna, 55.2% to 60.9% for Terra, and 64.4% to 73.6% for Sol, while Sol's best-of-two success (pass@2) moved only 1.2 points — meaning gains came from making already-reachable solutions repeatable rather than solving new tasks.
Is the improvement just from interrupting the agent to check its work?
The authors argue no. In a randomized five-arm experiment, a sham policy firing at identical moments on identical tasks but carrying generic text scored 35.7% versus 60.7% for the real policies on 14 eligible tasks, and generic instructions to verify or reconsider the solution (39-43%) also failed to reproduce the gain, suggesting the specific corrective content matters.
Who wrote this paper and is there a conflict of interest?
Both authors are co-founders of Failproof AI, the company that builds the runtime-policy infrastructure evaluated in the paper. This competing interest is disclosed in the paper, and the policy source, run-selection records, and per-attempt outcomes are released publicly on Hugging Face for independent inspection.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Extracted text omits most of §5.2–5.4 (funnel details, screen and complete-suite sections), so Discussion figures 70.7%/75.3% cannot be verified against their source tables
- Figures 1 and 3 are available only as captions; per-task arm-level results beyond quoted counts are not checkable
- Appendices B–H are partially truncated; power analysis and routing-audit details are not fully visible
- GPT-5.6 tiers (Luna/Terra/Sol) are provider routes, not immutable checkpoints — external readers cannot map them to public model artifacts
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.