Improving Information Extraction with Learned Queries
Omar Sharif, Soroush Vosoughi, Nikhil Singh
cs.CL
Quick verdict
Claims question design beats model scaling in LLM IE; oracle-conditioned +18.6 F1 is convincing, but deployable gains are modest and untested statistically.
Summary
The paper argues that in LLM-based information extraction, the bottleneck lies not only in model scale but in how models are queried. Across four clinical benchmarks (CaseReportBench, PHEE, DiscourseEE, MACCROBAT) and five prediction LLMs (Qwen3-4B/8B, GPT-OSS-120B, GPT-5-mini, Gemini-3.1-Pro), the authors present LoQ, a two-part framework. FeedQ iteratively optimizes document- and role-specific question sets: it generates questions, extracts arguments, scores them against ground truth, and refines via structured feedback, with an LLM leakage checker that rewrites questions that would reveal answers. The 12,820 optimized (document, role, question-set) triples then supervise LoRA fine-tuning of Qwen3-4B/8B into deployable question generators. On dev sets, oracle-optimized questions beat expert-derived questions by 18.6 F1 on average. On held-out test data, fine-tuned 4B/8B generators reach 62.7 mean F1, marginally above expert questions (61.3) and ahead of untuned larger generators (Gemini-3.1-Pro 60.1, GPT-5.2 54.9, GPT-OSS-120B 45.7). Out-of-domain results are mixed: +7.4 F1 on DocEE and +1.3 on MUC4, but a 5.4-point drop on GENEVA. Key limitations: no significance testing or seed variance, an LLM judge identical to the optimizer's backbone, and clinical-only primary benchmarks.
Review complete
Assessment at a glance
5
risk flags identified
● high 0
● medium 3
● low 2
5 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- Question design, not model scale, is the dominant bottleneck in LLM-based argument extraction: document-specific optimized questions raise F1 by 18.6 points on average over expert-derived questions across four clinical benchmarks and five prediction LLMs (dev sets, ground-truth-informed optimization).
- FeedQ, a feedback-driven loop with an LLM leakage checker, can automatically produce such question sets, with leakage flagged in 42-55% of iterations pre-filter and a 0.83% false-negative rate in a 600-set manual audit.
- Fine-tuning Qwen3-4B/8B on 12,820 optimized triples yields deployable question generators that reach 62.7 mean test F1, slightly above expert questions (61.3) and above untuned larger generators (Gemini-3.1-Pro 60.1, GPT-5.2 54.9, GPT-OSS-120B 45.7).
- The learned question-generation behavior transfers across all five prediction models and partially out of domain (+7.4 F1 DocEE, +1.3 MUC4, -5.4 GENEVA).
Strengths
Clean isolation of the question variable
Table 1 holds the prediction model fixed and varies only the question strategy across six conditions (No-Question, Knowledge-Q, CoT-Q, Contextual-Q, Zs-FeedQ, Opt-FeedQ); Table 2's precision-recall breakdown then explains mechanistically why naive Contextual-Q fails (DiscourseEE precision 37.4 to 28.8, recall 65.7 to 76.3).
Serious, quantified leakage handling
Appendix B.2 reports pre-filter ground-truth leakage in 42-55% of optimization iterations, a manual audit of 600 filtered question sets with a 0.83% false-negative rate, and an out-of-domain transfer argument (DocEE +7.4 F1) against implicit answer encoding.
Unusually thorough ablations
Practical deployment story
Internal numerical consistency
Weaknesses
Headline number is oracle-conditioned
No statistical rigor
Judge-optimizer circularity
Thin prompt-optimization baselines
Clinical-only core evidence with weak OOD coverage
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
The paper targets generative argument extraction with LLMs, framed as question answering: given a document and a target role (e.g., Adverse Drug Reaction), a model returns free-text arguments. The central claim is that the questions used to query the model - conventionally fixed, role-level templates from annotation guidelines - are a first-order bottleneck comparable to or larger than model scale. Supporting claims: (1) across four clinical benchmarks and five prediction LLMs, question optimization informed by ground truth yields +18.6 F1 on average over expert-derived questions (Table 1, dev sets; per-dataset range 11.0-32.7); (2) the FeedQ loop automates this with leakage safeguards; (3) fine-tuned 4B/8B question generators transfer the gains to unseen documents without ground-truth access, reaching 62.7 mean test F1 - above expert questions (61.3) and untuned larger generators (Table 5). A 12,820-triple dataset release rounds out the contribution.
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 5 strengths & 5 weaknesses, analyzed in detail
- 5 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What is FeedQ in simple terms?
FeedQ is an automated loop that improves the questions asked to an LLM during information extraction. For each document and target role, it generates candidate questions, has the LLM extract answers, compares them with ground-truth annotations, and rewrites the questions based on what was missed or hallucinated, for up to five rounds. A separate leakage checker removes any question that would give the answer away.
How much does question design improve extraction performance?
When questions are optimized with access to ground-truth annotations, the paper reports an average gain of 18.6 F1 points over expert-written questions across four clinical datasets and five LLMs (dev sets). The deployable version, a small fine-tuned question generator with no ground-truth access, reaches 62.7 mean F1 on test sets, about 1.4 points above expert questions and above untuned generators based on much larger models.
Do the learned questions leak the answers into the prompts?
The authors guard against this explicitly: an LLM-based leakage checker flags and rewrites questions containing ground-truth content. Before filtering, 42-55% of optimization iterations contained leakage; a manual audit of 600 filtered question sets found a 0.83% missed-leakage rate, and out-of-domain transfer (e.g., +7.4 F1 on DocEE) is offered as evidence the gains are not answer-encoding artifacts.
Does the method work outside clinical text?
Only partially. On three out-of-domain datasets (500 samples each), fine-tuned question generators improved DocEE by +7.4 F1 and MUC4 by +1.3, but hurt GENEVA by -5.4 F1, where multiple coreferent annotations favor the broader question sets of the untuned base model. All four primary benchmarks in the paper are clinical or biomedical.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Per-model table cells (Tables 1, 4, 7, 11) were extracted as flattened text, so cell-to-column attribution relies on header order; however, every row/column aggregate cited in this review was re-derived from the cells and matched the paper's stated means.
- The provided text ends mid-phrase inside the last cell of Table 12 (Appendix D, PHEE cluster descriptions of question edits); any content that followed that auto-interpretability table is not visible in the extraction.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.