TruthInsightBench: An Evidence-Grounded Benchmark for Automated Evaluation of Open-Ended Scientific Discovery Agents
Zhibo Yang, Chen Zhang, Yuewei Zhang, Hao Wang
cs.AI · cs.CL
Quick verdict
Discovery-oriented 40-task benchmark scored by an automated LLM-judge rubric; four agents on one base model plateau at 58-60/100; judge remains unvalidated.
Summary
TruthInsightBench is a benchmark designed to evaluate open-ended scientific discovery rather than reproduction. Its 40 blind tasks, drawn from 40 peer-reviewed studies (2018-2024) across 10 domains, expose only a neutral research goal and frozen data; source conclusions, expected values, and evaluation assets are withheld. A frozen LLM judge (GLM-5.1, quantized) scores the evidentiary maturity of each agent-formed claim on six dimensions operationalized as 29 artifact-grounded items, with deterministic aggregation and no per-instance human grading; task scores weight the top two de-duplicated discoveries 0.60/0.20 plus bounded yield, reference-coverage, and error bonuses. Four coding agents (Claude Code, OpenScience, Codex CLI, DeepSeek Harness) sharing one frozen base model (DeepSeek-V4-Flash) were each run once on all 40 tasks. Means form a narrow plateau (58.4-60.3/100) with no pairwise separation (Wilcoxon with Holm correction; TOST equivalence at ±4). Diagnostically, agents score 78-81% on evidence auditability and 83-87% on novelty but only 9-16% on robustness and control testing and near 0% on cross-dataset generalization, locating the deficit in scientific judgment rather than coding. Key limitations: the judge lacks human calibration or consistency audits, each system-task cell is a single run with no seed variance, agent run configurations are not publicly released, and blindness is only protocol-level given possible pretraining contamination.
Review complete
Assessment at a glance
5
risk flags identified
● high 2
● medium 3
● low 0
5 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- Reproduction-configured benchmarks conflate executing a prescribed analysis with making a discovery, and TruthInsightBench reconfigures task, evaluation object, and scoring to target discovery directly.
- Four coding agents sharing one frozen base model plateau at 58.4-60.3/100 with no statistically reliable pairwise separation under a single frozen run.
- The deficit concentrates in discriminating scientific acts — control testing (11-16%), robustness (9-14%), falsifiability (29-33%), cross-dataset generalization (near 0%) — while evidence auditability (78-81%) and novelty (83-87%) are strong.
- Automated, artifact-grounded scoring without per-instance human grading can supply the feedback signal for self-improving research loops.
- The bottleneck of current coding agents as AI scientists is scientific judgment rather than coding.
Strengths
Genuine design differentiation from reproduction benchmarks
Rather than re-tuning thresholds, the paper changes task setting, evaluation object, and scoring (Sec. 1); Table 1 positions ~20 benchmarks (PaperBench, DiscoveryBench, FIRE-Bench, ResearchClawBench, etc.) on five design axes and states complementarity explicitly rather than claiming superiority.
Auditable information-boundary and construction protocol
Sec. 3.1 formalizes the task tuple (g, D, M, T0 | p*, F*, A*); Sec. 3.2 specifies three admission conditions independent of evaluated systems' scores and 8 construction-time verifications per task (320 total), including re-derivation of two main results and six perturbation analyses; blindness is claimed only at protocol level.
Statistically careful plateau claim
Actionable diagnostic structure with trap tasks
Full per-task transparency
Weaknesses
Unvalidated judge underwrites every headline number
Single run per system-task cell
Execution layer irreproducible
Cross-dataset dimension partly structural zero
Novelty scoring under-specified and contamination unbounded
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
TruthInsightBench (arXiv 2609.05079v2) is a benchmark paper targeting a specific evaluation gap: existing scientific-agent benchmarks (PaperBench, CORE-Bench, DiscoveryBench, FIRE-Bench, ResearchClawBench) configure tasks, data, and rubrics around a hidden target study and reward recovery, which the authors argue measures execution rather than discovery. The benchmark reconfigures all three components: 40 blind tasks from 40 peer-reviewed studies (2018-2024; 34 of 40 from Nature Communications) across 10 domains expose only a neutral research goal, frozen data, metadata, and a literature cutoff T0; the source paper, reference findings, and evaluation assets (the tuple terms after the bar in Sec. 3.1) are hidden. The evaluation object is the evidentiary maturity of the agent's own claims, scored on six dimensions and 29 items (evidence auditability 45 pts, robustness 15, control testing 15, cross-dataset generalization 10, novelty 10, falsifiability 5) by a frozen LLM judge with deterministic aggregation. Central claims: (1) four coding agents on one frozen base model form a statistically indistinguishable plateau (58.4-60.3/100); (2) the deficit concentrates in discriminating acts (controls, robustness, falsifiability, cross-dataset validation), not coding; (3) the automated score can support self-improvement loops.
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 5 strengths & 5 weaknesses, analyzed in detail
- 5 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What is TruthInsightBench?
A benchmark of 40 blind, open-ended scientific data-analysis tasks derived from 40 peer-reviewed studies across 10 domains. Agents receive only a neutral research goal and frozen data, and an automated LLM judge scores the evidentiary quality of the scientific claims the agents form, along six dimensions and 29 artifact-grounded items.
How well do coding agents perform on TruthInsightBench?
Four coding agents (Claude Code, OpenScience, Codex CLI, DeepSeek Harness) sharing one base model scored 58.4-60.3 out of 100, statistically indistinguishable from each other. They are strong on evidence traceability (78-81%) and novelty (83-87%) but weak on control testing (11-16%), robustness (9-14%), falsifiability (29-33%), and cross-dataset validation (near 0%).
Is the TruthInsightBench code and data public?
The task data, frozen datasets, and the scoring program are released via the paper's GitHub repository (TruthInsight-stack/TruthInsightBench). However, the per-agent run configurations (prompts, decoding parameters, resource limits) and the original run artifacts are not redistributed, so the exact four-agent study cannot be re-run from the public package.
What are the main criticisms of this paper?
All scores come from a single LLM judge with no human calibration or consistency audit yet, and each system-task combination was run only once, so absolute numbers and the 'no difference between agents' conclusion need judge-validation and multi-seed replication before being treated as settled.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.