Annotated Surrogate Retrieval for Polish Statutory Law
cs.CL
Quick verdict
Statistically rigorous study of LM-annotated surrogates for Polish statute retrieval; 72.3% rank-1 win credible but confounded by gold-only surrogate coverage.
Summary
The paper introduces document surrogates — language-model annotations (summary, theme, concept set, hypothetical questions) attached to statutory articles at index time — and builds three retrieval designs for Polish statutory law: ASCR, a two-stage surrogate cascade with one listwise LLM reranking call; ASCR-H, which fuses a dense candidate list into the cascade; and DTF, which replaces pre-generation model calls with three lexical/dense retrievers, weighted reciprocal rank fusion, and a deterministic act-scoping prior. Evaluation uses 300 questions from the 2024–2025 Polish advocate and legal counsel entrance exams (264 with the reference article inside an 82,508-article corpus) against 14 baselines and 4 controls. ASCR-H leads at rank one with 72.3% (BM25 61.7%, dense 52.3%, DTF 51.9%), significantly ahead in 18 of 20 paired comparisons — behind only the oracle and not separated from its own no-concepts ablation (p=0.082). The advantage disappears by cutoff ten, where DTF, at one-ninth the latency, matches it and reaches the 70.3% oracle ceiling on citation accuracy. Reranking alone contributes 27.6 Hit@1 points. Key limitations, largely self-disclosed: surrogate annotations cover 27% of the corpus but 100% of reference provisions (an unquantified confound); DTF's act-scoping prior is never ablated; configurations ran once with hand-set hyperparameters; and answer accuracy is saturated.
Review complete
Assessment at a glance
5
risk flags identified
● high 1
● medium 3
● low 1
5 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- ASCR-H places the reference provision at rank one for 72.3% of retrievable questions (191/264), significantly more often than every non-oracle configuration except its own no-concepts ablation (18 of 20 comparisons significant, Table 3).
- The rank-one advantage does not survive depth: ASCR-H vs DTF is significant at k=1 and k=5 but not from k=10 onward, where DTF leads on point estimate at one-ninth the latency (Section 5.2, Figure 3).
- Listwise LLM reranking is the dominant component, worth 27.6 Hit@1 points (72.3% falls to 44.7% when removed, Table 4), contradicting predictions from Polish reranker generalisation studies.
- Ranking gains do not transfer downstream: ASCR-H citation accuracy (67.3%) is statistically indistinguishable from BM25 and from DTF, which matches the 70.3% oracle ceiling (Section 6.4).
- Lemmatisation, pseudo-relevance feedback and query rewriting each fail to improve retrieval on this benchmark (Section 6.3).
Strengths
Exemplary statistical protocol
Stratified McNemar with the Edwards continuity correction, per-comparison discordant counts (Table 3), Wilson intervals (Table 2), and a Holm–Bonferroni sensitivity check under which all rank-one conclusions survive (largest significant p=0.004, Section 3.3).
Transparency about the central confound
Sections 3.2 and Limitations state that surrogates cover 27.0% of articles but 100% of reference provisions, admit an unknown share of the advantage may stem from this, and name the missing bounding experiment (surrogate coverage of ranked non-gold candidates).
Depth-honest framing
Clean isolation of harmful context
Negative results and cost accounting
Weaknesses
Headline confound unquantified
DTF's decisive term never ablated
Single runs and hand-set hyperparameters
Downstream value undemonstrated
Missing document-expansion baseline
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
- All 5 strengths & 5 weaknesses, analyzed in detail
- 5 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What does the paper claim about retrieving Polish statutory articles?
It proposes three retrieval systems built on language-model annotations of statutes. The strongest, ASCR-H, places the governing article first for 72.3% of 264 bar-exam questions, versus 61.7% for BM25 and 52.3% for dense retrieval, with paired McNemar significance tests supporting the rank-one advantage (18 of 20 comparisons).
Does better retrieval improve the final legal answers in this RAG setting?
No measurable downstream gain was demonstrated. The test model answered 92.7% of exam questions correctly with no retrieval at all, and citation accuracy of the best-ranked system (67.3%) was statistically indistinguishable from simple BM25; the cheaper DTF system matched the 70.3% ceiling obtained by feeding the model the correct article directly.
What is the biggest limitation of this study?
Surrogate annotations exist for only 27% of the corpus but for 100% of the benchmark's reference articles, so systems matching against annotations may partly benefit from that asymmetry rather than from their design. The authors disclose this openly and did not quantify it.
Is the benchmark and data public?
Yes. The paper states that parsed questions, per-question outputs for every configuration, and all paired significance tests are available in the authors' public GitHub repository (OryCore/Research).
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Polish diacritics are corrupted by PDF extraction (e.g., 'Gr˛ebowiec', 'Yi˘git'); names were interpreted from context — an extraction artifact, not a paper defect.
- The metadata abstract is cut mid-sentence, but the complete abstract is present in the embedded full text; the embedded text runs through the references (page 16) with no signs of truncation.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.