MTAC-IFBench: Benchmarking Instruction-Following in Multi-Turn Agentic Coding
Bosi Wen, Cunxiang Wang, Jiayi Gui, Haoke Zhang, Yilin Niu, Pei Ke, Dayong Yang, Hongning Wang, et al.
cs.CL
Quick verdict
MTAC-IFBench: a 100-instance benchmark for multi-turn agentic-coding instruction-following; solid taxonomy and checks, but small, judge-conflicted, no CIs.
Summary
MTAC-IFBench is a benchmark for evaluating instruction-following by LLM code agents in multi-turn software development. Each of its 100 instances contains a repository policy file plus progressive user instructions averaging 7.04 turns and 91.33 constraints, drawn from a taxonomy of 6 primary and 18 secondary constraint categories. Instructions were synthesized by expanding single-turn tasks with Gemini-3.1-Pro, integrating constraints with Seed-2.0-Pro, and filtering through 17 human annotators. Evaluation pairs per-turn constraint checklists and a final function checklist with generated verification scripts and judge agents (Claude-Sonnet-4.6 for functions, DeepSeek-V4-Pro for constraints). Testing 11 LLMs under the Claude Code harness (8 also under OpenCode), the authors find the best model, GLM-5.2, satisfies 80.4% of constraints but achieves full per-turn compliance in only 12.7% of turns; most models approach zero strict compliance after six turns, and constraint compliance decays faster than functional correctness (Claude-Opus-4.6: 8.7% C-ISR vs 37.0% F-ISR). Static constraints are handled well, while orchestration, paradigm, and quantity constraints fail often. Main limitations: modest scale, no variance estimates or significance tests, functional results reported for only 6 of 11 models with GLM-5.2 absent, and judge models that are themselves evaluated systems — a self-preference risk the authors acknowledge but do not quantify.
Review complete
Assessment at a glance
4
risk flags identified
● high 0
● medium 3
● low 1
4 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- Existing benchmarks leave multi-turn instruction-following in agentic coding unevaluated; MTAC-IFBench fills this with 100 instances averaging 7.04 turns and 91.33 constraints across a 6-primary/18-secondary taxonomy.
- All 11 evaluated LLM-based code agents degrade rapidly in constraint compliance as sessions lengthen, with strict per-turn compliance (C-ISR) near zero after roughly six turns for most models.
- Perfect process-constraint compliance is far harder than functional correctness (best C-ISR 12.7% vs best reported F-ISR 37.0%), so outcome-only evaluation misses critical process failures.
- Agents handle static constraints (encoding, file paths) well but fail on orchestration, paradigm, and quantity constraints; policy-file constraints are more robust than user-instruction constraints, and replacing constraints is harder than adding them.
- The hybrid script-plus-judge verification agrees with human assessment on over 92% of constraint items and 85.5% of function items (Table 4).
Strengths
Well-motivated gap with a systematic constraint taxonomy
Section 3.1 grounds 6 primary / 18 secondary categories (Content, Language, Quantity, Style, Environment, Workflow) in instructions from a real platform serving over one million daily users plus five prior IF benchmarks, filling a genuine hole between outcome-only coding benchmarks (SWE-Bench, Terminal-Bench) and single-turn IF benchmarks (CodeIF, AgentIF), as positioned in Table 1.
Hybrid verification with a human-validated reliability audit
Deterministic constraints get generated Python scripts; subjective ones get judge agents with Playwright browser interaction (Sec 3.3, App. G). Table 4 shows 92.1-92.4% agreement with humans on 400 constraint items and 85.5% on 200 function items — above the bar for LLM-judged benchmarks, with N-F1 honestly reported.
Unusually transparent construction pipeline
Rich experimental matrix and actionable analyses
Weaknesses
Degradation claim is confounded by accumulating constraint load
Judge-evaluatee circularity
Incomplete functional-correctness results
No uncertainty quantification
Dataset selection and contamination opacity
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
MTAC-IFBench (arXiv 2609.14992) targets a real evaluation gap: instruction-following by LLM code agents in multi-turn development sessions. The paper advances three claims: (1) a 100-instance benchmark with progressive instructions (avg 7.04 turns) carrying 9,133 constraints (avg 91.33 per instance, 12.97 per turn) across a 6-primary/18-secondary taxonomy (Sec. 3.1); (2) a hybrid script-plus-judge verification protocol validated against humans (Sec. 3.3, Table 4); (3) empirical findings that constraint compliance collapses over long sessions and lags functional correctness (Sec. 4.3). The motivation is credible — SWE-Bench-style outcome metrics (Sec. 2) genuinely ignore process compliance, and prior IF benchmarks are single-turn (CodeIF, AgentIF) or lack a taxonomy (OctoBench), as the Table 1 comparison argues.
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 4 strengths & 5 weaknesses, analyzed in detail
- 4 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What is MTAC-IFBench?
A public benchmark of 100 multi-turn software-development tasks (average 7.04 turns and 91.33 constraints per task) designed to test whether LLM coding agents follow process constraints — coding style, file rules, workflow requirements — and not just produce working code. Code is on GitHub (abelperry/AgentProbe) and data on Hugging Face (thu-coai/MTAC-IFBench).
Which model performs best on MTAC-IFBench?
Under the Claude Code harness, GLM-5.2 leads with 80.4% of constraints satisfied on average (CSR), but every constraint in a turn is satisfied in only 12.7% of turns (C-ISR). Claude-Opus-4.6 follows at 78.6% CSR / 8.7% C-ISR. Most models drop to near-zero strict per-turn compliance after six turns.
Why is multi-turn instruction-following hard for coding agents?
Constraints persist across turns, so a late-turn response must satisfy dozens of accumulated rules at once. The benchmark shows agents lose track of earlier requirements — especially when constraints are updated or replaced — so strict all-constraints compliance collapses as sessions grow longer.
How are constraints verified in MTAC-IFBench?
Each constraint becomes a checklist item verified either by a generated Python script (for objective rules like keywords or line counts) or by a judge agent that inspects the project and interaction trace (Claude-Sonnet-4.6 judges functions with Playwright browser testing; DeepSeek-V4-Pro judges constraints). Manual spot-checks found about 92% agreement with human judgments on constraint items.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Figures 3-7 (constraint distribution, harness comparison, category/source/placement analyses) are image-only in the extraction; their plotted values could not be independently checked, only the in-text claims about them.
- Appendix prompt tables (8-18) may be compressed or garbled by text extraction; prompts were treated as described but not verified verbatim.
- Several cited works are 2026 preprints (e.g., OctoBench, SWE-Together, Terminal-Bench) that could not be cross-checked offline.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.