ProCredit: From Outcome Rewards to Progress Credit in Agentic Reinforcement Learning
Ming Ma, Yi Zhu, Yiran Zhong, Feida Zhu, Feida Zhu, Chonghan Liu, Pengkun Jiao, Pengkun Jiao, et al.
cs.LG · cs.CL
Quick verdict
Turns verifiable acceptance checks into per-turn progress rewards for agentic RL; consistent 3-scale AppWorld gains, clean ablations, but verifier-bound scope.
Summary
The paper targets a known weakness of outcome-reward RL for long-horizon LLM agents: a terminal reward discards how much progress failed attempts made and when, and groups with no successful trajectory produce zero gradient. ProCredit reruns each task's executable acceptance checks after every training turn, defines progress as the fraction of checks passed, and rewards each turn by its change in progress. This signal feeds two credit channels: a trajectory-level advantage that replaces the outcome with the score R + c(ΦT − Φ0) in GRPO's group standardization, and a turn-level advantage based on undiscounted onward returns centered by a pooled mean. Training Qwen3.5 models at 4B/9B/35B on 90 AppWorld tasks, ProCredit attains the highest task goal completion at every scale on both test sets — e.g., 71.53% versus 67.46% for the strongest outcome-only baseline (DAPO) at 4B test-normal — beats GRPO on ToolSandbox at all three scales, and leaves BFCL function-calling scores stable. Ablations show that adding progress to the trajectory score alone (GRPO-Φ) or spreading it over tokens does not help; the gain comes from crediting progress at the turn where it occurs, and γ=0.95 discounting hurts through a 'length tax'. Limitations: the method requires accessible, rerunnable environment checks; results rest on three seeds without significance tests; the reward coefficient c=0.5 is not ablated; and no code release is stated.
Review complete
Assessment at a glance
5
risk flags identified
● high 0
● medium 3
● low 2
5 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- Acceptance checks rerun on intermediate states yield verifiable per-turn progress signals without learned reward models, LLM-scored checklists, or reference trajectories.
- The performance gain comes specifically from turn-level credit for progress: augmenting the trajectory score alone (GRPO-Φ) or spreading rewards over tokens does not improve over GRPO.
- ProCredit achieves the highest task goal completion at all three Qwen3.5 scales on both AppWorld test sets (e.g., +4.1 TGC over the strongest outcome-only baseline at 4B test-normal) and beats GRPO on ToolSandbox at every scale.
- Discounting with γ=0.95 acts as a length tax that shortens trajectories and removes hard-task capability; undiscounted returns (γ=1) are better at all scales.
Strengths
Clean mechanism isolation via ablations
GRPO-Φ shares ProCredit's exact trajectory score but lacks the turn-level term and does not beat GRPO (65.40 vs 66.32 TGC at 4B test-normal), while ProCredit beats GRPO-Φ by 6.1/3.4/2.2 TGC at 4B/9B/35B; ProCredit(token) loses 8.3 TGC at 4B — pinpointing turn-level credit as the operative ingredient (Table 3).
Honest, self-contained theory
Appendix G proves score ordering (Prop 2) and policy invariance under solvability (Prop 3), explicitly noting the deviation from Ng et al.'s zero-terminal-potential condition; Appendix C's length-tax proposition is backed by measured credit statistics (shortest trajectory wins in 94.1% of 4B all-success groups at γ=0.95 vs 5.0% at γ=1).
Breadth and completeness of evaluation
Mechanistic discounting analysis
Candor about negative cases
Weaknesses
Narrow applicability gate
Small training regime
Thin inferential statistics
Unablated reward coefficient and baseline tuning parity
No code or data release stated
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
ProCredit targets credit assignment in RL for long-horizon LLM agents (AppWorld-style multi-app tool use), where a terminal outcome reward discards *how much* progress failed attempts made and *when* it occurred, and all-fail groups yield zero gradient. The enabling observation (§1, §3.2) is that the executable acceptance checks defining success can be rerun on intermediate environment states, so progress — the fraction of checks passed, Φ(s) — is as verifiable as the outcome itself. ProCredit rewards each turn by its change in Φ and uses the signal twice (§3.3): a trajectory-level advantage that replaces R with the score S = R + c(ΦT − Φ0) in GRPO's within-group standardization (Eq. 6), and a turn-level advantage built from undiscounted onward returns (γ=1) centered by a pooled mean over all turns of the group (Eq. 9). The claims are precise: (1) the gain comes from crediting progress at the turn where it occurs, not from enriching trajectory scores; (2) highest task goal completion (TGC) at three Qwen3.5 scales on both AppWorld test sets; (3) γ<1 discounting is actively harmful via a formalized 'length tax' (Prop 1, Appendix C).
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 5 strengths & 5 weaknesses, analyzed in detail
- 5 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What is ProCredit and how does it differ from GRPO?
ProCredit is a reinforcement learning method for LLM agents. GRPO gives an entire trajectory one reward at the end and shares it across all turns, so failed attempts look identical and all-fail groups give no learning signal. ProCredit instead reruns the task's acceptance checks after every turn during training, treats the fraction of checks passed as progress, rewards each turn by how much progress it added, and uses this both to compare whole attempts and to give the specific turn that advanced the task extra credit.
How much better is ProCredit than outcome-reward RL on AppWorld?
On AppWorld's test-normal set with Qwen3.5-4B, ProCredit reaches 71.53% task goal completion versus 67.46% for the strongest outcome-only baseline (DAPO) — a 4.1 point gain. It is the best method at all three model scales (4B/9B/35B) on both test sets, though margins shrink to 1–3 points at 9B and 35B, and DAPO scores higher on one secondary metric (SGC at 35B test-normal).
Why does reward discounting (γ < 1) hurt multi-turn agent RL?
The paper's 'length tax' argument: with γ=0.95, every extra turn shrinks the credit of all earlier turns by 5%, including the terminal success reward, regardless of what the extra turn accomplishes. So in groups where all trajectories succeed, the shortest one almost always gets the most credit (94% of cases measured), teaching the model to act in fewer turns and lose the accuracy that medium and hard tasks require. With γ=1 there is no length penalty and performance is better at all three model scales.
When can ProCredit's progress reward be applied?
It needs an environment whose state is accessible during training and whose success checks can be rerun on intermediate states without disturbing the episode — for example AppWorld's unit tests, ToolSandbox milestones, coding test suites, or database-state checks. If a task has only one or two checks, or passes all checks only on the final turn, the method collapses back to GRPO-style trajectory comparison. It cannot be applied where success is judged by a human or a learned model alone.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- All figures (Figs 1-3, A1-A3, C1, E1-E2) are available only as extracted captions and axis text; visual plot shapes (e.g., Fig 3 progress curves) could not be verified.
- The metadata abstract is truncated mid-sentence; the full abstract was recovered from the embedded paper body.
- Appendix D's Table D1 is garbled by extraction; the variant-to-configuration mapping was reconstructed from the surrounding prose.
- No supplementary material, repo links, or license statements appear in the extracted text.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.