MORSE: Multi-Context Ordering via Reverse Scoring for Evidence-Preserving Compression
Ke Wan, Yifan Wang, Liheng Lai, Chen Chen
cs.CL
Quick verdict
Shows context order changes what LLM compression keeps; MORSE's reverse-scored anchor plus search raises evidence retention in all 24 settings; QA gains mixed.
Summary
The paper studies context ordering as a decision variable in query-conditioned, likelihood-based extractive compression for multi-context RAG. Because sentence importance scores condition on previously serialized contexts, permutations of the same context set yield different deletion decisions. The authors attribute this sensitivity to 'information preemption': an earlier, partially relevant context can absorb credit for shared information, depressing the incremental score of a later, stronger evidence carrier. A controlled pair-swap intervention over 399 preemptor-evidence pairs shows evidence-first ordering rescues far more supporting units than it harms (e.g., 170 vs 16 at 4× compression). MORSE ranks contexts by reverse query likelihood (a PMI-style criterion), uses this as an evidence-first anchor, then evaluates K=5 candidate permutations (anchor plus four random) through the actual compressor and selects the output maximizing the reverse query evidence of the retained text. Across HotpotQA and 2WikiMultiHopQA (native and TRUE-20), 1P and iterative ISD compressors, and 4×/8×/16× budgets, MORSE attains the best Supporting-Fact Recall in all 24 settings, beating compute-matched RandomSearch-5 everywhere; gains transfer to a second scorer and to MuSiQue, though under the joint scorer-dataset shift the advantage over RandomSearch-5 is statistically inconclusive. Downstream QA improvements are less uniform, especially under ISD. Main limitations: small evaluation subsets (100-500 examples), a 0.5B likelihood scorer, roughly 9× search latency, and retention gains that do not always translate into F1/EM.
Review complete
Assessment at a glance
4
risk flags identified
● high 0
● medium 2
● low 2
5 strengths · 5 weaknesses · 0 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- Different permutations of the same retrieved context set produce markedly different evidence retention under an unchanged likelihood-based compressor and budget (Figure 1a).
- Information preemption — earlier partially relevant contexts absorbing credit for shared information — is the mechanism behind ordering sensitivity, directly supported by controlled pair-swap interventions on 399 pairs (Section 6.7).
- MORSE, combining a reverse query-likelihood anchor with compression-aware K=5 selection, achieves the best Supporting-Fact Recall in all 24 tested dataset-setting-procedure-ratio configurations and beats compute-matched RandomSearch-5 in every one (Tables 7-14).
- Ordering matters primarily through the compression stage rather than downstream presentation: compression-order changes add +9.58 SF-R on average while presentation-order changes add only +0.05 F1 (Section 6.6, Table 5).
- The Reverse anchor's advantage over random search transfers under individual scorer or dataset shifts but becomes statistically inconclusive under the joint OLMo-MuSiQue shift (+0.72 points, CI [-0.83, +2.34], Table 21).
Strengths
Causal mechanism test
Section 6.7 changes only the relative order of measured preemptor-evidence pairs (399 pairs): evidence-first placement rescues 170/165/149 supporting units vs harming 16/8/7 at 4×/8×/16×, and rescue rate rises monotonically with suppression quintile (24.1% to 48.8% at 4×).
Compute- and objective-matched baseline
RandomSearch-5 shares candidate permutations and the J_B selector with MORSE (A.3), isolating the anchor; the selector ablation adds a random-only pool control where J_B still leads (Spearman 0.397 vs 0.229 for Forward, A.6), ruling out anchor-in-pool artifacts.
Clean stage decomposition
Unusually consistent primary result
Calibrated negative results
Weaknesses
Proxy-metric gap
Small fixed subsets
Latency cost vs marginal gain
Same-scorer circularity
Compressor-family generality
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
The paper targets query-conditioned, likelihood-based extractive compression (LLMLingua-style) for multi-hop RAG inputs, promoting context ordering from a presentation detail to a decision variable inside the compressor. Three claims: (i) permutations of the same context set yield markedly different Supporting-Fact Recall (SF-R) under a fixed compressor, budget, and scorer (Figure 1a: 200 random permutations on HotpotQA); (ii) the mechanism is information preemption — an earlier partially relevant context absorbs credit for shared information and suppresses the incremental score of later, stronger evidence (Sections 4.1, 6.7); (iii) MORSE — a reverse query-likelihood anchor plus compression-aware selection among K=5 orderings — improves evidence retention over static Reverse and compute-matched RandomSearch-5 across 24 dataset-setting-procedure-ratio configurations, with downstream QA gains that are real but explicitly less uniform.
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 5 strengths & 5 weaknesses, analyzed in detail
- 4 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 0 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What is information preemption in LLM context compression?
When contexts are scored sequentially, an earlier passage that is only partially relevant can make shared information predictable, so a later passage carrying the actual answer evidence receives a low incremental score and is deleted first under the token budget. The paper demonstrates this with controlled pair swaps on 399 preemptor-evidence pairs: placing evidence first rescued 170 supporting units versus 16 harmed at 4x compression.
What does the MORSE method do?
It ranks each retrieved context by reverse query likelihood (roughly the PMI between context and query) to build an evidence-first anchor, then compresses K=5 candidate orderings (the anchor plus four random permutations) with the actual compressor and selects the output whose retained text has the highest reverse query evidence for the query.
Does better evidence retention improve answer accuracy?
Not always. Supporting-Fact Recall improved in every tested configuration, and F1/EM broadly improved under the one-pass compressor, but under iterative sentence deletion the downstream gains are mixed (e.g., Forward ordering beats MORSE on Exact Match in several ISD cells), and in some shifted settings (OLMo scorer on MuSiQue at 16x) SP-R rises while F1 falls.
How much extra compute does MORSE need compared to simple ordering?
Roughly 9x the latency of plain reverse ordering at K=5 (1.89 vs 0.21 seconds per example on five NVIDIA A6000 GPUs), rising to about 3.6 seconds at K=10. The extra cost yields diminishing returns: +4.5 SF-R points going from K=1 to K=5 at 4x compression, plus 2.4 more from K=5 to K=10.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Figures 1-2 are available only as text extraction; visual details (error bars, axes) are taken from captions.
- Some appendix content appears truncated in extraction; a few claims rest on in-text summaries rather than full tables.
- GitHub repository not inspected; code availability is as stated by the authors.
- 'Chain-QMI' is never explicitly defined in the extracted text; assumed to be the Eq. 1-2 sequential score s(u|q,h).
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.