Bioinfoysis Technical Report
Qingyang Shao, Xin Zhang, Zhouyang Yuan, Xianying Chen, Yujia Xiang, Zihao Yang, Tong Ye, Yangqi Zhang, et al.
cs.AI · cs.MA
Quick verdict
Multi-agent bioinformatics harness claims 82.4% BixBench SOTA with strong ablations; abstract-body numeric conflicts and audit-informed skills raise caution.
Summary
Bioinfoysis is a multi-agent harness for long-horizon bioinformatics analysis that treats each request as a persistent, artifact-grounded run rather than a transient chat. A planner maintains an executable checklist and revises pending steps from structured handoffs that bind each intermediate result to its responsible agent, checklist step, and plan generation; a controlled runtime validates generated scripts, tables, and figures, while role-specific context, persistent memory, and 78 governed bioinformatics skills support execution. On BixBench the system reports 82.44% accuracy (169/205) with Kimi-K2.6, ahead of the second-ranked system by 7.3 points, and 77.07-79.02% with DeepSeek-V4-Pro and GPT-5.4. On LAB-Bench 2, paired base-vs-agent comparisons across four LLMs raise the macro-average from 41.65% to 69.69% on SeqQA2 and from 3.13% to 27.60% on DbQA2. Ablations attribute the largest losses to removing persistent memory (-22.44 points) and all skills (-36.59 points). The authors also audit BixBench, documenting ambiguous counting units, environment drift, and data defects, and release a revised dataset. Main limitations: the abstract's SeqQA2/DbQA2 numbers contradict Table 2, no confidence intervals or significance tests are reported, baselines come from prior reports under different configurations, and the harness code is not released.
Review complete
Assessment at a glance
6
risk flags identified
● high 2
● medium 3
● low 1
4 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- Bioinfoysis achieves 82.44% (169/205) accuracy on BixBench with Kimi-K2.6, claimed 7.3 points above the second-ranked system (GPT Rosalind).
- The harness improves all four tested LLMs on LAB-Bench 2, raising macro-average accuracy from 41.65% to 69.69% on SeqQA2 and from 3.13% to 27.60% on DbQA2 (Table 2).
- Component ablations (Table 3) identify persistent memory (-22.44 pp) and governed skills (-36.59 pp with zero skills) as the largest contributors.
- BixBench contains material benchmark-quality defects — ambiguous counting units, environment drift, packaged-data mismatches — documented in a released audit dataset.
Strengths
Cross-model consistency
Table 3: 77.07-82.44% across Kimi-K2.6, GPT-5.4, and DeepSeek-V4-Pro with the full harness (5.37-point spread); Table 2 shows positive paired gains for all four LLMs on both LAB-Bench 2 tracks.
Complete ablation program
Table 3 quantifies each component: memory -22.44 pp, all skills -36.59 pp, domain skills alone -19.02 pp, fixed Plan-Execute -6.83 pp — unusual depth for a technical report.
Genuinely useful benchmark audit
Substantive systems design
Weaknesses
Abstract contradicts the body
Circular benchmark refinement
No statistical uncertainty
Second-hand, unmatched baselines
No code release and an accounting gap
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
Bioinfoysis (arXiv:2609.03871v2, DeepAutonomy team) is a technical report describing a multi-agent harness for long-horizon bioinformatics analysis. Central claims: (1) persistent, artifact-grounded runs with checklist-based planning and evidence-driven replanning achieve state-of-the-art BixBench accuracy of 82.44% (169/205) with Kimi-K2.6 (§4.2.1, Fig. 3), 7.3 points above the second-ranked GPT Rosalind; (2) the harness transfers across models — Table 2 reports paired base-vs-agent macro-averages rising from 41.65% to 69.69% on SeqQA2 and from 3.13% to 27.60% on DbQA2 across Kimi-K2.6, GPT-5.5, DeepSeek, and GLM-5.2; (3) ablations (Table 3) attribute the largest contributions to persistent memory (-22.44 pp) and governed skills (-36.59 pp with zero skills); (4) BixBench itself contains benchmark-quality defects, documented in a released audit dataset (Appendix B.1, HuggingFace Bixbench-Verify).
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 4 strengths & 5 weaknesses, analyzed in detail
- 6 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What is Bioinfoysis and how accurate is it on BixBench?
Bioinfoysis is a multi-agent system ('harness') for long-horizon bioinformatics analysis: it plans with an executable checklist, binds every intermediate result to an agent, checklist step, and plan version, validates generated files, and uses persistent memory plus 78 curated bioinformatics skills. Its technical report claims 82.44% accuracy (169/205 questions) on the BixBench benchmark using Kimi-K2.6 as the underlying model, which would lead the reported leaderboard by 7.3 percentage points.
Is Bioinfoysis code or data available?
As of this report, no code for the harness itself is released. The authors publish a demo website and a HuggingFace dataset (ZhangxinMath/Bixbench-Verify) containing their audit of BixBench questions, with revised questions and reference answers.
What problems did the authors find in the BixBench benchmark?
Their audit found ambiguous counting units (e.g., whether 'variants' means table rows or unique variant calls), hidden sample-filtering rules, results dependent on software or database versions, mismatched packaged data (a Zebra Finch chromosome-length file identical to Jackdaw's), and reference notebooks whose logic contradicts the question text. They argue some apparent agent 'errors' are actually defensible alternative analyses.
Which LLMs does Bioinfoysis support and how much does the harness help?
The report evaluates Kimi-K2.6, GPT-5.4/5.5, DeepSeek-V4, and GLM-5.2. On LAB-Bench 2's SeqQA2, paired comparisons raise the four-model macro-average from 41.65% (base models alone) to 69.69% (with the harness); on DbQA2, from 3.13% to 27.60%. Gains are largest for the weaker base models, e.g., DbQA2 base accuracies of 0-7.8% rise to 20.9-37.2%.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Figures 2-4 (architecture diagram, BixBench leaderboard, SeqQA2 heatmap) are raster images; only their captions were extracted, so leaderboard entries beyond the values quoted in prose could not be checked.
- The extraction ends cleanly at the end of Appendix B.2 (p. 28) with a complete final sentence; no truncation of body text was observed.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.