A Framework for Generating Valid Context-Specific Benchmarks through Expert Guidance
Kimberly Le Truong, Nari Johnson, Anna Kawakami, Hoda Heidari
cs.AI · cs.CY
Quick verdict
Schema-guided synthetic benchmark generation with expert input; validity metrics + case study show realism/coverage gains, but evidence is single-domain.
Summary
This paper presents an end-to-end framework for building context-specific LLM evaluation datasets that combine structured expert input with synthetic generation, targeting the usual tradeoff between expert-designed benchmarks (high validity, high cost) and purely synthetic ones (scalable, but often unrealistic or off-scope). The authors introduce a schema—decomposed from population, concept, and instance into eight fields plus seed examples—to elicit deployment context from domain experts, and define four dataset-quality criteria grounded in measurement validity: coverage, diversity, content realism, and stylistic realism, each with a concrete metric (Cartesian-product coverage with a smooth penalty, DCScore, anchored Sinkhorn distance, style-embedding distance). In a case study with U.S. school social workers, schema-guided generation (Claude Sonnet 4.6, 3 seeds, 100 examples) outperformed a few-shot baseline on content realism (0.77 vs 0.73) and coverage (0.23 vs 0.16), matched diversity, and scored lower on automated stylistic realism (0.56 vs 0.60) even though five of six interviewed experts preferred its style and content (Likert 4.2 vs 3.0). Ablations over 1,013 schema configurations show which fields matter: systematized instance and constituents drive coverage, deployment population boosts content realism (+11.87%), while seed examples slightly reduce coverage (−3.44%). Limitations include a single case study, n=100 datasets, LLM-judge labels with low baseline agreement (0.30), and no statistical testing.
Review complete
Assessment at a glance
5
risk flags identified
● high 0
● medium 3
● low 2
4 strengths · 4 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- A lightweight schema (population/concept/instance decomposed into eight fields plus seed examples) elicits the minimum expert context needed to guide synthetic generation of context-specific evaluation datasets.
- Four metrics — coverage, diversity, content realism, stylistic realism — operationalize content and ecological validity for evaluation datasets without requiring a ground-truth reference distribution.
- Schema-guided generation beats a few-shot baseline on content realism (0.77 vs 0.73) and coverage (0.23 vs 0.16) with 5/6 expert preference, while diversity is unchanged and automated stylistic realism is slightly lower.
- Ablations over 1,013 configurations show which expert inputs matter most: systematized instance and constituents for coverage, deployment population for content realism, and seed examples can actually reduce coverage.
Strengths
Validity theory made operational
§4.2 and Appendix A map content validity to coverage+diversity and ecological validity to content/stylistic realism, converting abstract 'representativeness' (Bean et al., 2025) into computable formulas: Cartesian-product group coverage with a smooth penalty (Eq. 1), anchored Sinkhorn OT distance (Eq. 2), style-embedding distance (Eq. 3).
Honest reporting of metric-expert divergence
§6 openly reports that the automated stylistic realism metric contradicts expert preference (0.56 baseline-favorable vs Likert 4.2 schema-favorable), using it to diagnose the seed-representativeness assumption rather than hiding it.
Unusually thorough ablations
Robustness and sensitivity work
Weaknesses
Low absolute dataset quality
LLM-judge dependency undermines the headline coverage contrast
Single case study with structural-only generalization
Rater-schema overlap and no inferential statistics
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
The paper attacks the validity–scalability tradeoff in benchmark construction: expert-built evaluation sets are high-quality but slow, while synthetic generation scales but yields unrealistic or redundant data. Its three claims are (1) a schema for eliciting minimum expert context (population, concept, instance → eight fields, plus seed examples, §4.1), (2) four quantitative dataset-quality criteria grounded in measurement validity (coverage, diversity, content realism, stylistic realism, §4.2/App. A), and (3) that schema-guided synthetic generation improves dataset quality over a few-shot baseline. The empirical vehicle is a U.S. school social work organization (~30 staff) evaluating LLM-generated reflective questions, building on Kawakami et al. (2026), whose workshops produced the 16 expert-written seed examples.
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 4 strengths & 4 weaknesses, analyzed in detail
- 5 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What does this paper propose?
An end-to-end framework for creating LLM evaluation datasets tailored to a specific deployment context: domain experts fill a structured schema (population, concept, instance — eight fields plus seed examples), which then guides an LLM to synthetically generate benchmark examples. Quality is assessed with four measurement-validity-based metrics: coverage, diversity, content realism, and stylistic realism.
How much better is schema-guided generation than ordinary few-shot generation?
In the social-work case study (Claude Sonnet 4.6 generator, 3 seed examples, 100 generated examples), the schema method scored higher on content realism (0.77 vs 0.73) and coverage (0.23 vs 0.16), tied on diversity, and slightly lower on the automated stylistic realism metric (0.56 vs 0.60). Five of six interviewed domain experts preferred its style and content (Likert 4.2 vs 3.0 for style).
Is code or data available?
The implementation is released on GitHub (KimberlyTruong/expert-informed-eval-gen), and generation prompts plus the empty schema are in the paper's appendices. However, the 16 seed examples come from anonymized prior work and are not published, so exact reported numbers cannot be fully re-derived without equivalent seeds.
Does the framework work outside social work?
Structurally, yes: Appendix H shows the schema accommodates a clinical-transcription use case and a political-misinformation example without field changes, and maps two existing evaluation frameworks onto it. But these are illustrative demonstrations only — no empirical validation in a second domain has been done yet.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Several passages were omitted by text extraction (parts of §4.2, Appendix B.2, codebook Table 3, Appendix I); judgments rely on the surrounding intact text.
- Figure 2's per-field contribution values were extracted as a garbled numeric strip; only values restated in prose (+11.87%, +12.81%, −3.44%, ~9% drop) are treated as reliable.
- Likert data are n=6 with no dispersion per participant reported beyond Table 2, limiting statistical assessment.
- Seed examples are anonymized, so reproduction of exact metric values is impossible even with the released code.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.