When Learned Context Planning Fails to Beat Strong Retrieval: A Controlled Study of Planning, Routing, and Reranking for Long-Context QA
cs.CL · cs.IR
Quick verdict
A controlled negative-result study showing SFT evidence planners lose to strong retrieval on LongBench-v2; rigorous controls but narrow scope, weak planner.
Summary
The paper asks whether learned context planning — an SFT-trained selector that picks evidence atoms before answering — can beat strong retrieval controls on long-context multiple-choice QA. Using all 503 LongBench-v2 questions with Qwen2.5-7B-Instruct, the authors train a LoRA planner on 140 outcome-selected traces (28 dev), explicitly noting the 503-question primary diagnostic is partly transductive; a disjoint 152-question test split is reported separately. At the 18k-character budget, anchored hybrid retrieval reaches 36.18% accuracy and BM25 35.98%, versus 34.19% for the best planner-guided method; on the untouched test split anchored hybrid leads 42.11% to 36.84%. Leakage-safe routers fail to convert a large oracle gap (46.52% hybrid/planner oracle; 56.26% all-method oracle). Under tight budgets the planner leads by only +0.40 points at 6k (p=0.930) and loses 1.19 points at 9k; planner-guided reranking shows +1.79 points at 6k with the paired interval crossing zero. Packing-order and score-flatness analyses yield nothing after Benjamini-Hochberg correction. The paper concludes learned planning is a weak relevance signal, not a retrieval replacement. Main limitations: a single benchmark, one 7B answer model, a planner trained on only 168 questions, MiniLM-scale retrievers and rerankers, and a disclosed audit discrepancy (262 versus 168 retained traces) with no bitwise-reproduction claim.
Review complete
Assessment at a glance
4
risk flags identified
● high 0
● medium 2
● low 2
4 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 17 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- At the 18k-character budget on all 503 LongBench-v2 questions, anchored hybrid retrieval (36.18%) and BM25 (35.98%) beat the best direct planner-guided method (Planner+hybrid anchor, 34.19%).
- Large oracle routing gaps (46.52% for hybrid/planner, 56.26% for all methods) do not convert into deployable gains for leakage-safe routers, which stay at or below 36.38%.
- Under tight budgets, planner gains shrink or vanish: +0.40 points at 6k (p=0.930), −1.19 at 9k, and planner-guided reranking's +1.79 at 6k has a paired interval crossing zero.
- Packing-order and retrieval-score-flatness analyses find no stable mechanism, with none of 92 win-loss tests surviving Benjamini-Hochberg correction.
- In this setup, learned context planning behaves as a weak relevance feature rather than a replacement for strong retrieval.
Strengths
Unusually strong control ladder
§2/§3.1 and Tables 1/4 compare the planner against BM25, MiniLM dense, RRF hybrid, anchored hybrid, budgeted selectors (weighted/no-anchor packing, MMR), and a ms-marco-MiniLM cross-encoder — controls most planner papers omit, which is precisely the paper's thesis.
Conservative, well-documented statistics
Exact two-sided sign tests with paired wins/losses, Wilson CIs (Table 1), paired bootstrap for the held-out split ([−12.50, +1.97]; §3.1), BH correction across 92 mechanism tests (§3.4), and a power analysis (~8,800 examples needed for +1.5 points; §4). The borderline p=0.061 (Planner+BM25 vs BM25) is correctly not claimed significant.
Leakage discipline and honest transductivity disclosure
Artifact transparency including self-correction
Weaknesses
Negative result may reflect planner data starvation, not planning's ceiling
'Strong retrieval' is modest by current standards
Primary analysis is partly transductive and held-out power is thin
Reproducibility caveats on the historical run
Narrow external validity
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
The paper is a controlled negative-result study asking whether learned context planning — an SFT-trained model that selects evidence atoms before the answer model reasons — survives strong retrieval, routing, budgeted-selector, and reranking controls on LongBench-v2 multiple-choice QA. Its central claims (§1, §4): (i) direct planner-guided construction loses to BM25 and anchored hybrid retrieval; (ii) oracle complementarity does not become a deployable router; (iii) low-budget planner gains vanish under stronger selectors; (iv) the remaining rerank signal is statistically unsupported. The scope is deliberately narrow and stated honestly in Limitations: one benchmark, one 7B answer model, one planner recipe. The question is whether the title's generality matches that scope.
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 4 strengths & 5 weaknesses, analyzed in detail
- 4 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What does this paper conclude about learned context planning versus retrieval?
On LongBench-v2 multiple-choice QA with Qwen2.5-7B-Instruct, an SFT-trained evidence-selection planner did not robustly beat BM25, anchored hybrid retrieval, budgeted selectors, or cross-encoder reranking. The authors conclude that, in this setup, learned planning acts as a weak relevance signal rather than a replacement for strong retrieval.
Was the comparison fair, given the planner was trained on some of the test questions?
Partially. The main 503-question analysis includes 168 questions used in planner training (partly transductive), which the authors disclose. They also report a disjoint 152-question test split with no training overlap, where retrieval still leads 42.11% versus 36.84%.
What benchmark, model, and planner were used?
All 503 multiple-choice questions of LongBench-v2, with Qwen2.5-7B-Instruct as the answer model. The planner is a LoRA adapter (rank 16, 3 epochs) fine-tuned on 140 reward-selected planning traces plus 28 development traces mined from the base model's own sampled plans.
Are the paper's code and artifacts available?
Yes. The authors release evaluation and analysis code, split IDs, checksums, text-free per-example predictions, atom IDs, retrieval and reranker ranks/scores, and paired-test outputs in a public GitHub repository. Base model weights, adapter weights, and raw LongBench-v2 contexts are not redistributed.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Tables are embedded as inline extracted text; cell alignment for Table 4 and the domain table relies on extraction order, though all cited numbers were cross-checked against surrounding prose.
- No figures appear in the extracted text; abstract through references appear complete, so little content is assumed missing.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.