PetQA: Benchmarking Veterinary Knowledge and Clinical Reasoning
Taegyun Kim, Youngwook Ham, Jungwook Rhim, Ju-Hyun An, Sungkyu Park, Kunwoo Park
cs.CL · cs.AI · cs.CV
Quick verdict
Claims first long-form veterinary QA benchmark (18.8K Korean pairs, 18 models); rigorous dataset, but LLM-judge metrics show weak human correlation.
Summary
PetQA is a Korean long-form question-answering benchmark for veterinary medicine, built from 18,827 real-world dog/cat Q&A pairs (10,076 text-only, 8,751 multimodal) scraped from Naver Knowledge iN, with answers from platform-verified veterinarians. The test split, PetQA-Bench (2,000 pairs per modality), adds question-type and clinical-condition labels annotated by six AI students under a Krippendorff's alpha>0.7 pilot protocol; a veterinary PhD judged 85/100 sampled reference answers correct. The authors benchmark eighteen models (closed/open LVLMs, open LLMs) under zero-shot, RAG, and SFT settings using ROUGE-L, BERTScore, and LLM-judge factuality/helpfulness. Closed models lead: GPT-4.1 mini tops text ROUGE (0.303), GPT-4.1 factuality (0.569), Gemini 2.5 Pro helpfulness (4.923); all models score worse on multimodal questions even after controlling for question-type mix; RAG and SFT yield inconsistent gains, with SFT inflating surface similarity while cutting helpfulness for all 14 fine-tuned models. The dataset is released with five translated versions. Key limitations: helpfulness judge-human correlation is weak (Pearson 0.2255) and factuality judge only ~0.41; 15% of expert references were judged incorrect (rankings nevertheless robust, Spearman 0.936-0.975); the RAG retrieval corpus is undisclosed; annotations were made by non-veterinarians.
Review complete
Assessment at a glance
5
risk flags identified
● high 0
● medium 3
● low 2
5 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- PetQA is, to the authors' knowledge, the first long-form QA dataset for veterinary medicine, comprising 18,827 real-world dog/cat QA pairs with expert veterinarian answers plus question-type and clinical-condition annotations.
- Closed models generally outperform open-weight models on veterinary QA, with the largest gaps appearing on LLM-as-a-judge factuality and helpfulness metrics.
- All eighteen models perform consistently worse on multimodal questions than on text-only questions, a gap that persists after controlling for question-type distribution (Table A6).
- RAG and SFT yield inconsistent improvements on judge-based metrics: SFT raises ROUGE/BERTScore for all fine-tuned models while reducing helpfulness for all 14 (text set) and factuality for 11 of 14, indicating style mimicry over factual gains.
Strengths
First-of-kind resource filling a real gap
Section 2/Table 1 show existing veterinary benchmarks are MCQA-only (Luo et al. 2025; Tam et al. 2026); PetQA's 18,827 long-form pairs (10,076 text, 8,751 multimodal) from real Naver Knowledge iN consultations with platform-verified veterinarian answers is a genuinely new evaluation target.
Unusually disciplined annotation and audit pipeline
Section 3.2: iterative guideline refinement until Krippendorff's alpha>0.7 on a 50-sample pilot, six trained annotators, a 5,378-entry controlled clinical-condition list, and a veterinary PhD audit of 100 references (99 relevant, 85 correct), plus IRB approval (SSU-202604-HR-805-1).
Statistical gating of claims is above the norm
Honest judge calibration reporting
Insightful adaptation analysis
Weaknesses
Headline conclusions rest on weakly validated judge metrics
Reference answers are imperfect and labels are made by non-veterinarians
RAG section is not reproducible
Generation constraints may cap long-form scores
Single platform, single language, two species
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
PetQA (arXiv:2609.04598) introduces a Korean long-form QA benchmark for veterinary medicine, built from real dog/cat questions on Naver Knowledge iN paired with answers from 23 platform-verified veterinarians: 10,076 text-only and 8,751 multimodal pairs (18,827 total; Table 1). The test split, PetQA-Bench (2,000 pairs per modality), carries question-type and clinical-condition labels. The paper makes three claims: the resource is the first long-form veterinary QA dataset; eighteen models benchmarked under zero-shot, RAG, and SFT show closed models leading, uniform multimodal degradation, and inconsistent adaptation gains; and five translated versions broaden applicability. The resource claim is well supported; the benchmarking conclusions are sound at the system level but rest partly on judge metrics with weak human validation.
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 5 strengths & 5 weaknesses, analyzed in detail
- 5 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What is PetQA?
PetQA is a Korean long-form question-answering benchmark for veterinary medicine. It contains 10,076 text-only and 8,751 multimodal (image + text) Q&A pairs about dogs and cats, taken from real consultations on the Korean platform Naver Knowledge iN, with answers from platform-verified veterinarians. Its test split, PetQA-Bench (2,000 pairs per modality), also carries question-type and clinical-condition labels, and translated versions exist in English, German, Chinese, Indonesian, and Arabic.
Which AI model performs best on veterinary questions according to PetQA?
Closed (proprietary) models lead overall in the paper's zero-shot benchmark: on the text set, GPT-4.1 mini had the highest ROUGE (0.303), GPT-4.1 the highest factuality (0.569), and Gemini 2.5 Pro the highest helpfulness (4.923); on multimodal questions Gemini 2.5 Pro led ROUGE (0.246) and helpfulness (4.814). Every model scored lower on image-based questions than on text-only ones.
Is the PetQA dataset publicly available and can I use it?
Yes. The authors release PetQA through their GitHub repository (ssu-humane/PetQA) under the CC BY-NC-ND 4.0 license, which allows non-commercial research use but prohibits commercial use and distribution of derivative works, in line with Naver Knowledge iN's terms of service.
How reliable are the LLM-as-a-judge scores in the PetQA paper?
Use them with caution. On 100-sample human validations, the factuality judge (Gemini 2.5 Flash) correlated with human ratings at Pearson 0.4065, while the helpfulness judge (GPT-4o) reached only 0.2255. The authors themselves flag the helpfulness metric as needing further validation, so cross-model gaps on the judge-based metrics are suggestive rather than definitive; the traditional metrics (ROUGE, BERTScore) are computed deterministically from the released data.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Figures 2, 3, 4 and A1 are rendered images; their plotted values (e.g., exact PM scores in Figure 4, per-question-type heatmap values in Figure 3) are not present in the extracted text, so numeric claims about them rely on the surrounding prose.
- The worked examples inside the judge-prompt figures (A6/A7) appear only as placeholders ({Example 1}...) in the extraction; the example content itself is not available as text.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.