arXiv paper2026-09-03· machine learning — reinforcement learning for LLM agents· method paper

TIGPO: Temporal Instance-Graph Policy Optimization for Long-Horizon LLM Agents

cs.LG

Quick verdict

TIGPO extends graph-based agent RL with persistent per-task graphs and revisit scheduling; gains over GraphGPO are modest and lack significance testing.

Summary

This paper proposes Temporal Instance-Graph Policy Optimization (TIGPO), a reinforcement-learning method for training long-horizon LLM agents that extends graph-based credit assignment across policy updates. Existing graph-based methods such as GraphGPO build state-transition graphs only from the current rollout batch and discard them afterwards. TIGPO instead maintains a persistent per-task transition graph, recomputes shortest-path distances over the merged graph, and uses historical transitions purely as detached structural references that never enter the policy loss. It splits a fixed rollout budget between Exploration slots for new tasks and Revisit slots that reattempt earlier tasks after a delay of ten updates, and pairs each Revisit group with the detached scores of its earlier Exploration group to stabilize advantage estimation under small rollout groups (four rollouts per group). On ALFWorld it reaches 91.28% overall success versus 91.02% for GiGPO and 89.32% for the reproduced GraphGPO; on WebShop it reaches a score of 88.65 and 77.54% success. Ablations show revisit scheduling alone hurts performance, while cross-temporal comparison recovers the loss. Main limitations: gains are small relative to seed variance, statistical significance is untested, ablations cover only ALFWorld, and no code is released.

Review complete

Assessment at a glance

machine learning — reinforcement learning for LLM agents

risk flags identified

● high 0

● medium 2

● low 2

Reproducibility readinessmedium

Code & data availablenot stated

Review confidencehigh

AI text signal0/20 passages flagged

Deep research viableCode-reproducibleNeeds GPU

4 strengths · 5 weaknesses · 5 author questions

AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.

Levels reflect verification complexity — not a quality score. Evolve never rates your paper.

Key claims under review

Maintaining a persistent per-task transition graph across policy updates improves step-level credit assignment for current rollouts while historical transitions are never replayed in the policy loss.
An Exploration-Revisit scheduler with a 10-update delay reconnects current rollouts with historical graph structure under a fixed budget, raising task-memory and exact-state hit rates (Figure 2).
Pairing each Revisit group with detached scores from its earlier Exploration group stabilizes small-group advantage estimation, adding 1.70 pp on ALFWorld over the revisit-only variant (Table 2).
TIGPO outperforms GiGPO and GraphGPO on ALFWorld (91.28% vs 91.02% and 89.32% overall success) and WebShop (88.65 score / 77.54% success vs 87.94/73.83% and 87.34/76.37%) at an equal 64-trajectory budget with no observable time or memory overhead (Tables 1, 3).

Strengths

Clean on-policy discipline for history reuse
Historical transitions and scores are used only as detached structural/statistical references (Eqs. 4-8); the training batch B(k) in Eq. 9 contains only current-policy rollouts, sidestepping the stale log-probability mismatch the paper correctly identifies in naive replay (§1, §2).
Honest, mechanism-level diagnostics
Figure 2 measures task-memory and exact-state hit rates, showing the scheduler actually reconnects rollouts with stored structure rather than assuming it; Table 3 interprets overlapping timing IQRs as 'comparable runtime' rather than claiming a speedup (247.08 vs 258.32 s/step).
Instructive ablation structure (Table 2)
Budget- and compute-matched comparison

Weaknesses

Group-size confound in the headline comparison
Sub-point margins without significance testing
Ablations cover only ALFWorld
Per-category regressions and duplicated statistics
Unspecified state identity and memory scaling

Full Review Report

$1.99 · one-time

Unlock the complete preliminary analysis Evolve produced for this paper.

Report preview

TIGPO: Temporal Instance-Graph Policy Optimization — Structured Review

Scope & Claims

A method paper (ICLR 2027 submission, single author, Nanjing University) proposing three coupled mechanisms for long-horizon LLM-agent RL: (1) persistent per-task transition graphs carried across policy updates (§4.1), (2) an Exploration–Revisit scheduler that splits a fixed budget of B task groups between new tasks and delayed reattempts (Δ = 10 updates) of old ones (§4.2), and (3) a cross-temporal advantage that normalizes each Revisit group against the detached scores of its paired earlier Exploration group (§4.3, Eq. 7). The central conceptual claim is that graph-based c

The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.

All 4 strengths & 5 weaknesses, analyzed in detail
4 risk flags with severity, section & reviewer questions
Methods & statistical evidence assessment
5 questions a peer reviewer would ask
Full six-dimension evaluation report & recommendations

One-time purchase · this paper only

One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.

Frequently asked questions

What is TIGPO in one sentence?

TIGPO is a reinforcement-learning method for LLM agents that keeps a persistent per-task state-transition graph across policy updates, schedules delayed revisits of previously attempted tasks, and compares current rollouts against detached earlier scores to improve credit assignment without ever replaying old trajectories in the loss.

Does TIGPO replay historical trajectories during training?

No. Historical transitions and scores are used only as detached structural references and statistics for computing advantages; every trajectory that receives gradients is freshly generated by the current policy, so the method remains on-policy despite reusing history.

How much better is TIGPO than GraphGPO and GiGPO?

On ALFWorld, TIGPO reports 91.28% overall success versus 89.32% for a locally reproduced GraphGPO (+1.96 points) and 91.02% for GiGPO (+0.26 points). On WebShop, it reports 77.54% success versus 76.37% and 73.83%, at the same 64-trajectory-per-update budget. No statistical significance tests are reported, and the GiGPO margin is within reported seed variation.

What model and benchmarks were used to evaluate TIGPO?

Qwen2.5-1.5B-Instruct trained for 250 policy updates on ALFWorld (text-based household tasks, 3,827 instances across six categories) and WebShop (web shopping), with the final checkpoint evaluated over 512 episodes per seed across three seeds; GPT-4o, Gemini-2.5-Pro, and prompting/RL baselines are included for context.

About this review

This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.

Figure 2's hit-rate curves appear in the extraction only as axis labels and captions; the plotted hit-rate values cannot be verified from the text.
The prose says results adopted from prior work are marked with † in the tables, but the dagger symbols are not visible in the extracted Table 1, so which baseline rows are reproduced versus adopted must be taken from the prose alone.

Go deeper on this paper

Wrote a paper? Stress-test it before reviewers do.

Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.

Scan your paper