Why Sample What You Can Enumerate? Exact Policy Optimization for Genomic Tool Selection
Haoyue Liu, Xiaoyu Ma, Ye Chen, Zhichao Wang, Xiaoying Tang
cs.AI
Quick verdict
FGPO replaces GRPO's sampled rollouts with exact enumeration of 16 tool subsets, winning all 15 genomic settings; convincing but limited to tiny action spaces.
Summary
This paper argues that sampled policy optimization (GRPO) becomes structurally mismatched when the tool-subset space is small enough to enumerate. In genomic QA with four tools (16 subsets), GRPO samples six rollouts per question; as the policy concentrates, sampled rewards collide and group-normalized advantages vanish — the dead-group rate rises from 0.2% of questions under a uniform policy to 20.8% after training (79.6% under the differential reward GRPO actually uses). The authors propose FGPO, which scores all 16 subsets, optimizes the exact action expectation, and precomputes an exhaustive question–subset reward table so frozen-reasoner calls leave the training loop. Across three genomic benchmarks and five frozen reasoners, FGPO beats GRPO in all 15 settings by 6.75 points on average (up to 14.20) while invoking fewer tools (1.40 vs 2.36 per question on GenomeQA); a controlled k-subset intervention shows accuracy rises monotonically with action coverage. Main limitations: exact enumeration applies only to very small action spaces, GRPO and FGPO train on different rewards, and single-run results lack seed variance.
Review complete
Assessment at a glance
4
risk flags identified
● high 0
● medium 2
● low 2
4 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- In enumerable tool-selection spaces, GRPO's sampled groups go dead as the policy concentrates: the dead-group rate rises from 0.2% of questions (uniform policy) to 20.8% at G=6 on GenomeQA after training, and 79.6% under the coarser differential reward GRPO trains on.
- FGPO optimizes the exact expectation over all 2^n tool subsets using a precomputed question–subset reward table, removing frozen-reasoner calls from the training loop entirely.
- FGPO outperforms GRPO in all 15 benchmark×reasoner cells by 6.75 points on average (up to 14.20) while invoking fewer tools (1.40 vs 2.36 per question on GenomeQA).
- A controlled k-subset intervention shows accuracy is strictly monotone in action coverage (49.58→52.59 for k=2→16), attributing the gain to coverage rather than confounds.
- A standard on-demand GRPO schedule would require 2.4× as many frozen-reasoner reward evaluations as FGPO's one-shot table construction.
Strengths
Mechanism identified with both closed-form diagnostic and intervention
Dead-group probability is derived in closed form (Eq. 7, Prop. 1 with proofs in F.1: coarsening and concentration both raise it), measured at 0.2%→20.8% at G=6 on GenomeQA (17.8%/17.4% on the held-out suites), and then tested causally by the k-subset dose–response (Table 3: peaks 49.58→50.47→51.39→52.59 for k=2→16; McNemar p=0.011 vs k=8).
Unusually thorough baseline fairness
GRPO is shown converged across all eight checkpoints plus four independent configs (43.90–45.57, Table 13); DPO's collapse is shown for every β and checkpoint (Table 14); anonymous-slot routing is verified by retraining with real descriptions (51.67 vs 51.75, B.1); sensitivity to β, softmax temperature and a 1.5B selector is reported (Table 5).
Honest ceiling and residual reporting
Transfer and cost results are practically meaningful
Weaknesses
Estimator–reward confound in the headline comparison
Regime narrowness
No seed variance for trained arms
Offline baselines may be under-specified
Genomic depth is thin
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
The paper (arXiv 2609.10221, cs.AI) claims a structural mismatch between GRPO-style sampled policy optimization and tool-selection problems whose action space — subsets of a small tool library — is enumerable. In the n=4 genomic setting (16 subsets), it claims: (i) sampled groups die as the policy concentrates (dead-group rate 0.2%→20.8% at G=6 on GenomeQA; 79.6% under the coarser differential reward GRPO actually trains on); (ii) FGPO fixes this by optimizing the exact expectation Σ_a qθ(a|s)·r(s,a) over all subsets, with an exhaustive precomputed reward table removing reasoner calls from training; (iii) FGPO beats GRPO in all 15 benchmark×reasoner cells by 6.75 points on average (max 14.20); and (iv) a k-subset dose–response attributes the gain to action-space coverage. Claims are scoped honestly to the small-library regime; scaling is explicitly deferred.
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 4 strengths & 5 weaknesses, analyzed in detail
- 4 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What is FGPO and how does it differ from GRPO?
FGPO (Full-Group Policy Optimization) is a training method for LLM tool selection. Instead of GRPO's sampled rollouts (six per question), it scores every possible tool subset — 16 for a four-tool library — and optimizes the exact average reward over all of them. This eliminates 'dead groups', where all sampled rewards coincide and the group-normalized advantage is zero, which the paper shows afflicts 20.8% of questions after GRPO training. It also precomputes a reward table once, so the frozen reasoner is never called during training.
Why do GRPO advantages vanish during training?
As GRPO's policy concentrates probability on preferred tool subsets, its sampled rollouts increasingly land in the same reward class; when all G sampled rewards are equal, every group-normalized advantage is zero and the group teaches nothing. The paper proves (Appendix F.1) this dead-group probability increases when reward classes merge or mass concentrates, and measures it rising during training alongside a 3.3× entropy collapse.
Does FGPO work with more than four tools?
The paper includes a five-tool stress test (32 subsets) where the policy learns to reject a harmful homology-search tool, calling it on only 2.8% of questions and beating exhaustive invocation (45.38 vs 44.51). But enumeration cost grows as 2^n, so the method is inherently limited to small libraries; extending it via adaptive candidate-set construction is left to future work.
What accuracy does FGPO reach on the genomic benchmarks?
On GenomeQA with the training reasoner (Qwen3-8B), FGPO reaches about 52.6%, versus roughly 45.4–45.5% for GRPO, 39.3% with no tools, and 49.4% invoking all tools. The per-question oracle is 77.4%, so substantial headroom remains. The same policy transfers without adaptation to four other frozen reasoners, with FGPO ahead of GRPO in all 15 benchmark–reasoner combinations.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- The provided text is a sampled excerpt with '[...sampling gap...]' markers (parts of §2.3–2.4, §3.6, Related Work §4, and appendix fragments were skipped); judgments about skipped content rely on surrounding context only.
- Figures are assessed from captions and in-text numbers; the rendered figures themselves were not verifiable from the extraction.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.