Arman Cohan

Research profile

Research summary

Arman Cohan is a researcher at Yale University whose recent work centers on evaluating large language models, particularly where AI meets science. The OpenAlex-based record — h-index 35, i10-index 75, 144 works in the last three years, and 62 distinct coauthors — shows sharp acceleration since 2023, peaking at 55 works in 2025. The four 2026 papers reviewed by Evolve form a coherent evaluation-methodology program: an expert re-grading audit finding that about 95% of apparent frontier-model failures on four physics benchmarks stem from benchmark or grader defects; BackTrend, a backward-reconstruction benchmark for scientific weak-signal prediction where systems reach only F1 ≤ 10.1%; IdeaAMBIG, showing LLMs detect just 9.6% of implementation-critical gaps in research-idea specifications yet clarify 80.6% once directed; and ActReview, which post-trains Qwen3-8B with rebuttal-derived data and rubric-based GRPO rewards for actionable peer review. The field distribution spreads evenly across four evaluation-oriented areas, and frequent coauthors span NLP and Yale physical-sciences groups. This profile is name-resolved from public bibliographic data tied to the listed ORCID.

Research keywords

Highlights

• Expert re-grading finds most LLM physics-benchmark failures are defects of the benchmarks themselves

  A 2026 expert audit of four physics benchmarks traces about 95% of apparent frontier-model failures to benchmark or grader defects rather than model errors — though the corrected scores rest on a new question set and grader.

• BackTrend reframes scientific foresight as backward reconstruction

  The 2026 benchmark evaluates weak-signal prediction in science by reconstructing trajectories backward; best systems reach only F1 ≤ 10.1%, which the authors partly attribute to strict matching and construction bias.

• IdeaAMBIG quantifies the specification gap in research ideas

  LLMs detect only 9.6% of implementation-critical gaps in research-idea specifications, but clarify 80.6% once the gap is identified — an evidence-grounded benchmark whose LLM-judged scoring still needs validation.

• ActReview turns rebuttals into training signal for actionable peer review

  Post-training Qwen3-8B on rebuttal-derived review feedback with rubric-based GRPO rewards improves actionability, though the judge (GPT-5.4) is embedded in its own pipeline.

Timeline

1. 2013Earliest substantive year in the OpenAlex record, with 1 work; growth is gradual through the 2010s (2 works in 2014, 3 in 2015).
2. 2017A steady ramp begins: 8 works in 2017, 9 in 2018, then 11 in each of 2019–2021.
3. 2023Output accelerates to 30 works, nearly 2.5 times the 12 recorded in 2022.
4. 202445 works recorded, continuing the steep climb.
5. 2025Peak so far at 55 works.
6. 202644 works already recorded; all four Evolve-reviewed papers — BackTrend, the physics-benchmark re-grading audit, IdeaAMBIG, and ActReview — appear this year.

Papers in the Evolve catalog

• 2026-09-21

  BackTrend: Evaluating Scientific Weak-Signal Prediction via Backward Reconstruction

• 2026-09-11

  How Good Are Frontier Models at Physics? Expert Re-Grading Reveals Broken Evaluations and Near-Saturation of Leading Benchmarks

• 2026-09-09

  IdeaAMBIG: Benchmarking Implementation-Critical Gaps in Research-Idea Specifications

• 2026-09-08

  ActReview: Rebuttal-Guided Training Data and Rubric Rewards for Actionable Peer Review Generation

Frequent co-authors