BIT.UA at BioASQ 14B: Modular Retrieval with pg_textsearch and Qdrant, and Agent-Based Answer Generation
André Ribeiro, Rúben Garrido, Alexander Christiansen, Richard A. A. Jonker, Sérgio Matos
cs.CL
Quick verdict
BIT.UA's BioASQ 14B system paper modernizes retrieval/generation infrastructure but its headline changes underperformed officially; candid negative results, no statistical testing.
Summary
This system paper describes BIT.UA's BioASQ Task B (14B) pipeline: BM25 moved to PostgreSQL pg_textsearch, dense retrieval added via Qdrant and BGE-M3, hybrid fusion (RRF or weighted sum), HyDE query expansion, the Context-1 agentic retriever, 29 newly trained cross-encoder rerankers with a curriculum ShifterSampler, plus LLM-as-judge and multi-agent quorum generation; snippets subtask entered for the first time. Internal validation on the 13B golden data shows hybrid retrieval lifting MRR from 0.549 to 0.696 and the ShifterSampler beating BasicSampler on all four architectures tested. Officially, however, the new components largely underperformed BM25-only baselines, with best Phase A MAP rank 5 (Batches 1, 3); quorum systems achieved perfect yes/no F1 in Batch 3 Phase A+, and small models proved competitive at roughly ten times lower cost. Key limitations: validation reuses the 13B golden data also used for tuning; no statistical significance testing; some rerankers reached validation MAP 0.9995 amid suspected data leakage and failed officially; several results were confounded by infrastructure problems; the authors themselves concede Phase A performance regressed versus prior editions.
Review complete
Assessment at a glance
6
risk flags identified
● high 3
● medium 1
● low 2
4 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- Migrating BM25 to PostgreSQL pg_textsearch and dense retrieval to Qdrant/TEI improved operational reliability, with no index-corruption or synchronization issues across all four batches (Section 5.1).
- Hybrid first-stage retrieval outperforms BM25 alone in internal validation (Table 2: MRR 0.549 to 0.696 with RRF; 0.738 with weighted sum plus BGE-reranker-v2-m3), but official results did not consistently confirm this advantage.
- The agent quorum mechanism delivered the strongest generation results, including perfect yes/no F1 (rank 1) in Batch 3 Phase A+ (Table 7) and the best factoid MRR of 0.5000 (Phase B, Batches 3–4, Table 8).
- Small-model quorums (Gemma-4-E2B/E4B, Nemotron-3-Nano-4B) are competitive with much larger and proprietary configurations at roughly a tenth of the inference cost (~4 vs ~17 model calls per question).
- Curriculum-based hard-negative sampling (ShifterSampler) consistently improves reranker MAP over uniform negatives across all four tested architectures (Table 4, gains +0.0052 to +0.0445).
Strengths
Operational infrastructure gains are real and honestly framed
The pg_textsearch + Qdrant + TEI migration (Sections 3.2.1–3.2.2) is justified by reliability, not speed: Section 5.1 reports no index-corruption or synchronization issues across all four batches, and BM25 remained the backbone of every submission.
Comprehensive internal retrieval validation with clear ablations
Table 2 spans BM25, two fusion methods, two rerankers and Context-1 across five metrics; Table 4 isolates the sampler with identical architectures/epochs/data, showing ShifterSampler beats BasicSampler on all four architectures (+0.0052 PubMedBERT-base to +0.0445 MiniLM-L-6-v2).
Unusually candid negative-result reporting
The agent quorum is specified reproducibly and cost-analyzed
Weaknesses
Validation set doubles as the tuning set, and it diverges from official results
Zero statistical rigor throughout
Internal contradictions between prose and tables
Known-leaky component still shipped in ensembles
Headline claims overstate outcomes
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Preliminary Review — BIT.UA at BioASQ 14B (arXiv 2609.04999)
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 4 strengths & 5 weaknesses, analyzed in detail
- 6 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What is the BIT.UA BioASQ 14B paper about?
It is a shared-task system report from University of Aveiro's BIT.UA team on BioASQ Task B round 14: how they rebuilt their biomedical question-answering pipeline using PostgreSQL pg_textsearch for BM25, Qdrant/BGE-M3 for dense retrieval, newly trained reranker models, and LLM-as-judge plus a multi-agent debate ('agent quorum') mechanism for answer generation, with results and lessons from all four test batches.
What is the 'agent quorum' mechanism in this paper?
A generation method where 3–5 LLM agents, each given a different 'thinking focus' (analytical, skeptical, evidence-based, etc.), debate a biomedical question over multiple rounds. Each agent sees a small adaptively retained document set, gives an opinion and an agreement level each round, and the debate stops on full consensus, a round cap, or stable disagreement; a synthesizer model then writes the final answer.
Did the new hybrid retrieval beat BM25 in the BioASQ challenge?
No. In internal validation hybrid retrieval improved MRR from 0.549 to 0.696, but in the official Phase A rankings hybrid submissions generally placed worse than BM25-only systems; the team's best official MAP rank was 5 (Batches 1 and 3), achieved with BM25 + DPRF-style setups. The paper candidly flags this validation-vs-official mismatch as unresolved.
What were the paper's best official results?
Phase A document retrieval: MAP rank 5 in Batches 1 and 3. Answer generation: perfect yes/no F1 (rank 1) in Batch 3 Phase A+ from two quorum systems, factoid MRR 0.5000 in Phase B Batches 3 and 4 (ranks 8 and 5), list F1 up to 0.5671 (rank 9, Phase B Batch 4), and yes/no 0.9352 (rank 3, Phase B Batch 4).
Is the BioASQ14b code available?
Yes. The abstract and paper state all code is openly available at https://github.com/bioinformatics-ua/BioASQ14b, and the appendices point to prompt template files and reranker training configurations in the repository.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Section 4.2.2 ends mid-sentence ('produced lower snippet MAP...') in the provided extraction; the sentence's continuation is not available, but the remainder of the paper (Sections 5–6, appendices) is present.
- Tables are reconstructed from PDF text extraction; cell-to-row alignment (e.g., Table 1 agent rosters, Tables 7–10 per-batch scores) could not be double-checked against the original PDF layout, so table-prose mismatch flags are based on the extracted values.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.