Coverage, Not Targeting: A Structural Regime in Multi-Turn Agent Credit Assignment
Chenyu Zhou, Qiliang Jiang, Shuning Wu, Xu Zhou
cs.LG · cs.AI
Quick verdict
Argues uniform credit spread beats targeting under terminal verifiers (low Vd); well-controlled and pre-registered, but real-model evidence is single-update.
Summary
The paper studies credit assignment in multi-turn agentic RL where a terminal verifier scores only the final environment state. It proposes verifier information density Vd = k/C — the fraction of the agent's causal tool chain whose per-turn correctness the verifier exposes — and argues terminal-state verifiers sit at Vd≈0.15 (τ²-bench: k=1 in 98% of rollouts, C=5–8), far below a synthetic crossover Vd*≈0.8 where targeting beats uniform spreading. In shared-rollout first-update comparisons on τ²-bench retail (Qwen3-14B, GRPO+LoRA), a continuous dense reward spread uniformly beats the sparse binary outcome reward (+0.079 assert-support; binary degrades the base on 4/5 seeds), while concentrating the same advantage on progress turns or random turns is equally harmful (per-turn −0.053; shuffled statistically indistinguishable from per-turn) — concentration, not targeting, is the harmful axis. The account predicts and confirms uniform wins on BFCL V3 (Vd≈0.4; shuffled 8/8 negative) and on ToolACE-2-8B (pooled Δ=−0.048 over 32 pre-registered seeds), with a matched-budget breadth sweep tracing a monotone dose–response that vanishes at full chain coverage and a reward-to-go arm reaching parity. A write-fraction ceiling plus a benchmark census (Table 3) generalizes the regime. Main limitations: real-model evidence covers only the first policy update; main-benchmark power is thin (5 seeds, 19–20 tasks, between-batch noise exceeding effect sizes); the cross-family geometry effect is shown on BFCL only, with τ² Qwen3-only.
Review complete
Assessment at a glance
5
risk flags identified
● high 0
● medium 2
● low 3
5 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- Terminal-state verifiers collapse per-turn credit to a single final-write turn (k=1 in 98% of measured rollouts) while success needs C=5–8 prerequisite steps, so real agents sit at Vd≈0.15, far below the synthetic crossover Vd*≈0.8.
- In this low-Vd regime, concentrating a fixed terminal advantage is equally harmful whether targeted at progress turns or random turns — coverage breadth, not credit location, is the first-order quantity.
- On τ²-bench retail the sparse binary outcome reward is net-harmful (−0.038 vs base, 4/5 seeds) while uniform dense redistribution improves the base (+0.041).
- A write-fraction bound caps non-oracle Vd for state-transition verifiers, and no common agentic benchmark exceeds Vd≈0.4 (BFCL V3), making the coverage regime the structurally common case.
- The effect reproduces across benchmarks and model families (BFCL V3; ToolACE-2-8B pooled Δ=−0.048 over 32 seeds), with a matched-budget breadth sweep tracing a monotone dose–response that vanishes at full chain coverage.
Strengths
Matched-concentration shuffled control cleanly isolates the harmful axis
Random-turn concentration lands within noise of progress-turn concentration (shuffled−per-turn +0.014, CI [−0.07, +0.10]) while both trail uniform (§4); the same structure reproduces on BFCL V3 (+0.002), directly falsifying 'wrong turns' as the per-turn deficit's explanation.
Mechanism triangulated across synthetic, measured, and census evidence
The MLP phase boundary (Vd*≈0.81, robust across C=4–10, widths 8–32, K=5–20, three algorithms), the real-rollout diagnostic (k=1 in 98%; data-flow provenance in 100% of successful terminal writes, §5.2), and the write-fraction census over 8 benchmarks (Table 3) all converge on the same account.
Ex-ante predictions confirmed and nulls integrated as predictions
Pre-registered replication ladder on a cross-family agent
Evaluation noise quantified and designed around
Weaknesses
Real-model evidence stops at the first policy update
Thin statistical power on the primary benchmark
Cross-family geometry effect shown only on BFCL V3
Crossover Vd*≈0.8 never observed on a real model
Key diagnostic sample sizes not fully stated
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
- All 5 strengths & 5 weaknesses, analyzed in detail
- 5 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What is verifier information density (Vd) in this paper?
Vd = k/C: the fraction of an agent's C-step causal tool-call chain whose per-turn correctness the verifier independently exposes. Terminal-state verifiers score only the final write (k=1), so measured Vd is ≈0.15 on τ²-bench and ≈0.4 on BFCL V3 — both far below the synthetic crossover Vd*≈0.8 where targeting beats spreading.
Why does uniform reward spreading beat per-turn credit assignment here?
Because under terminal verification the observable credit collapses to a single turn (k=1 in 98% of measured rollouts) while success needs a 5–8-step prerequisite chain. Any concentrated spike covers ~1 of 5–8 causal turns wherever it lands — the paper shows progress-targeted and random-turn concentration are equally harmful, so coverage breadth, not credit location, decides the outcome.
Which benchmarks and models show the effect?
The geometry effect is shown on τ²-bench retail (Qwen3-4B/8B/14B, GRPO+LoRA) and reproduced on BFCL V3 (Qwen3-8B and the Llama-family ToolACE-2-8B, 32 pooled seeds). A census (Table 3) argues the low-Vd terminal-verifier structure is shared by SWE-bench, WebArena, OSWorld, and AppWorld.
Is sparse binary outcome reward always harmful for agent RL?
Only in the studied setting: on τ²-bench retail, above a capability floor, in single-update comparisons, the binary reward degraded the base policy on 4/5 seeds. Below the floor (Qwen3-4B, Llama-3.1) no credit geometry had usable gradient, and the paper does not claim binary rewards are harmful in general RL domains.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- No truncation detected: the extraction runs complete from the abstract through reference [28], including Appendices A–G.
- Figure graphics are not present in the extraction, but every figure's key values (Fig. 2, 6, 7 deltas) are restated in body text or tables, so no analysis relied on the images.
- The paper references a 'released harness' but no repository URL appears in the extracted text; code availability is inferred from wording, not a link.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.