ORQA: An Occupation-Realistic Question and Answer Framework for LLM Professional Knowledge
Shreyas Krishnan, Serina Chang, Abhishek Nagaraj
cs.CL · cs.AI
Quick verdict
ORQA: automated pipeline yields a 480-item source-traceable occupational benchmark; scalable and honest, but 67% filter agreement and thin contamination power.
Summary
ORQA is a benchmark-construction method that converts authoritative occupation-specific web sources into multiple-choice tests of LLM professional knowledge. The pipeline assigns each O*NET occupation a whitelist of trusted domains (regulators, licensing bodies, professional societies), retrieves whitelisted documents, extracts evidence cards with GPT-4o, generates six-option questions whose correct answers must be entailed by a quoted passage, and applies nine automated quality checks plus human review; annotator-automation agreement on 'good' items is about 67%. The final bank holds 480 questions across 116 occupations, all 21 SOC major groups, and 187 source hosts. Evaluating 15 models closed-book with three seeds each, Claude Opus 4.6 leads at 62.4%, ahead of GPT-5.4 (60.3%) and Claude Sonnet 4.6 (58.7%); open-weight models score 33-41%. The frontier ordering persists under open-ended three-judge scoring and wage-bill weighting. Accuracy varies sharply by occupation: frontier models reach 78-80% on Healthcare Practitioners versus 36-43% on Office and Administrative Support, and GPT-5.4 scores 0% on Actuaries, Sheet Metal Workers, and Fish and Game Wardens. ORQA correlates with GDPval (Spearman +0.90) and GDPval-AA Elo (+0.92). Main limitations are the modest human-filter agreement, an underpowered contamination check, sparse coverage in some SOC groups, and LLM-judge-based open-ended scoring.
Review complete
Assessment at a glance
5
risk flags identified
● high 0
● medium 3
● low 2
5 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- An automated whitelist-plus-entailment pipeline can produce a scalable, source-traceable occupational QA benchmark (480 items, 116 occupations, 21 SOC groups, 187 hosts).
- Frontier models cluster at 58.7-62.4% closed-book with open-weight models at 33-41%, and the ordering is robust to open-ended judging and wage-bill weighting.
- Aggregate leaderboards mask large occupation-level variation, including 0% accuracy for GPT-5.4 on Actuaries, Sheet Metal Workers and Fish and Game Wardens.
- ORQA rankings correlate strongly with external signals (Spearman +0.90 with GDPval, +0.92 with GDPval-AA), supporting validity as a non-contradicted directional measure.
- Pre-training contamination does not appear to drive closed-book scores, since 14 of 15 models score equal or better on post-cutoff sources — a check the data can only weakly support.
Strengths
Source-traceable construction at scale
Per-occupation domain whitelists derived from O*NET, verbatim entailment-anchored answers, and 480 items spanning 116 occupations, all 21 SOC groups and 187 hosts; research-paper sources are excluded at fetch time (§3), directly addressing the digital-jobs bias the authors document in agent benchmarks.
Robustness of the headline ordering
The frontier trio ranks first in closed-book MCQ (62.4/60.3/58.7), open-ended three-judge scoring (54.2/53.1/50.6), and wage-bill weighting (66.8/64.2/63.4); SEs use an occupation-level cluster bootstrap, the correct dependence unit given within-occupation item similarity (§3).
Unusually honest validation posture
Actionable occupational heterogeneity
Release assets and external triangulation
Weaknesses
Quality gate validated by a single agreement statistic
No selection funnel reported
Difficulty pretest selection effect
Underpowered contamination analysis
LLM-only open-ended judging and fixed E/F options
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
ORQA (arXiv 2609.12366, UC Berkeley) proposes an automated pipeline that converts authoritative occupation-specific web sources — regulators, licensing boards, professional societies, government publications — into source-traceable QA items for benchmarking LLM professional knowledge. The authors position it between skill-mapped benchmarks and expensive expert-created ones (GDPval), targeting scalability, economy-wide coverage, and occupation-level assessment. Four contributions are claimed (§1): a scalable construction system; a 480-item bank over 116 occupations, all 21 SOC major groups, 187 hosts; a 15-model closed-book MCQ plus open-ended experiment with occupation-clustered uncertainty and wage-bill weighting; and external validation against GDPval, GDPval-AA and the Anthropic Economic Index. Scope is explicitly narrowed to source-traceable occupational knowledge (reporting responsibilities, thresholds, procedures, guidelines), not overall occupational capability — a framing the Discussion maintains honestly.
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 5 strengths & 5 weaknesses, analyzed in detail
- 5 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What is ORQA?
ORQA is a 480-question benchmark of LLM professional knowledge built automatically from authoritative occupation-specific sources — regulators, licensing boards, professional societies and government publications. Every correct answer is traceable to a quoted source passage, and the bank covers 116 US occupations across all 21 SOC major groups from 187 websites.
Which models perform best on ORQA?
Closed-book multiple choice: Claude Opus 4.6 (62.4%), GPT-5.4 (60.3%) and Claude Sonnet 4.6 (58.7%) lead; open-weight models score 33-41% versus a 16.7% random-guess baseline. With a web-search tool, frontier models rise to roughly 78-80% on multiple choice.
How does ORQA relate to GDPval?
ORQA accuracy correlates strongly with GDPval win rates (Spearman +0.90 across 5 overlapping models) and with the GDPval-AA Elo ranking (+0.92 across 11 models). Unlike GDPval, which relies on expert-written tasks, ORQA scales automatically — but it tests source-traceable knowledge rather than end-to-end task production.
Is ORQA contaminated by pretraining data?
The authors compare accuracy on sources published after each model's training cutoff: 14 of 15 models do as well or better on post-cutoff items, arguing against memorization. However, post-cutoff samples are very small (n=3 for GPT-5.4), 54% of items could not be dated, and the authors themselves call the check directional.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Figure payload data (Figures 2-17) extracted as jumbled numeric strings; all numbers cited in this review come from the narrative prose and Tables 1-4, which are intact.
- The abstract in the task metadata cuts off mid-sentence, but the full embedded paper text contains the complete abstract wording, so no content was lost.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.