Harbor Adapters and Harbor-Index: Infrastructure and a Curated Meta-Dataset for Large-Scale Agentic Evaluation
Lin Shi, Haowei Lin, Zixuan Zhu, Xiaoyue Zhou, Xiang Li, Xiangning Lin, Yaxuan Deng, Han Xu, et al.
cs.AI · cs.CL
Quick verdict
Unifies 80+ agentic benchmarks under one harness; $300K, 54-benchmark study yields an 82-task Harbor-Index (top score 28%). Credible, open infrastructure work.
Summary
The paper presents Harbor Adapters, an infrastructure layer that ports more than 80 agentic benchmarks onto the Harbor task framework (a shared instruction/environment/test/solution schema), reducing benchmark-agent integration from O(mn) to O(m+n). Adapters are validated through three-stage code review and multi-trial parity experiments against original harnesses. On this infrastructure the authors evaluate 8 models from the Gemini, Claude, and GPT families under 16 model-harness configurations (Terminus-2 plus one vendor-native harness each) across 54 benchmarks and 6,627 tasks, three trials per configuration (about 0.3M trajectories, 226B tokens, over $300K). Key findings: base-model capability outweighs harness design (fixed-effect ranges 0.451 vs 0.087); benchmark rankings are highly redundant (after selecting 12 benchmarks, remaining ones correlate at ρ≥0.7; three tasks per benchmark recover system rankings at ρ≈0.923); frontier models use fewer tokens yet cost 2-3× more per trial; and many unresolved tasks stem from task defects rather than difficulty. The third contribution, Harbor-Index, distills 82 audited tasks from 29 benchmarks via difficulty filtering, AI and human review, and an audit-and-fix loop; no configuration exceeds 30% pass rate (best: GPT-5.5 with Codex at 28.0%). Limitations include coverage of only 54 of the 80+ adapted benchmarks from three vendors, reliance on an LLM judge with moderate agreement (κ=0.51) for the 6,028-trajectory failure analysis, and some parity experiments run once or transitively.
Review complete
Assessment at a glance
6
risk flags identified
● high 0
● medium 3
● low 3
5 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- Harbor Adapters reduce benchmark-agent integration from O(mn) to O(m+n) while preserving original-benchmark semantics, as verified by three-stage review and parity experiments (mean±SEM).
- Base-model capability is a substantially larger driver of benchmark performance than harness design: fixed-effect ranges 0.451 vs 0.087 (5.2×), with 6/8 model vs 2/4 harness coefficients significant at p<0.05 under a benchmark-random-intercept LMM.
- The 54-benchmark space is highly redundant: after 12 greedy-selected benchmarks every remaining one correlates at ρ≥0.7, and 3 tasks per benchmark recover system rankings at mean ρ≈0.923.
- Frontier models use fewer tokens than weaker models at every empirical difficulty level (42% of tokens in the easiest bucket) yet cost 2-3× more per trial, with their accuracy advantage peaking in the 0.3-0.7 difficulty band.
- Harbor-Index (82 tasks, 29 benchmarks) is difficult and unsaturated: no model-harness configuration exceeds 30% pass rate, with the best (GPT-5.5 + Codex) at 28.0%, and failures decompose into 453 timeouts, 361 near-misses, and 445 far-off answers.
Strengths
Unusual scale with full artifact release
16 model-harness configurations over 54 benchmarks and 6,627 tasks, 3 trials each (~0.3M trajectories, 226B tokens, >$300K, §2); adapters, full trajectories, results, and Harbor-Index are open-sourced (GitHub, Harbor Hub, harbor-index.org), enabling independent re-analysis.
Adapter validation pipeline is rigorous and documented per benchmark
Three-stage review (bot → junior/senior human → lead sign-off, >10,000 GitHub comments, App. D.3.2) plus parity experiments with mean±SEM (App. D.4); Appendix D.6 documents oracle failures honestly (e.g., GSO 89/102 due to timing variance, SWE-Bench Pro 97% with 15 invalid gold patches, AA-LCR needing two ground-truth fixes).
Quantified model-vs-harness and redundancy analyses with explicit statistics
Honest treatment of task quality and its own limitations
Harbor-Index is designed against the failure modes the paper itself documents
Weaknesses
Model pool and version naming are inconsistent across sections
Native-harness arm is vendor-locked by design
Circularity in the difficulty-bucket efficiency analysis
Judge-based failure analysis has moderate reliability
Parity rigor is uneven across the adapter suite
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
The paper makes three claims. (1) **Infrastructure**: Harbor Adapters port 80+ agentic benchmarks onto the Harbor task schema (instruction/environment/test/solution; §2), cutting benchmark-agent integration from O(mn) to O(m+n); as of May 2026 the stack supports 22 agents and multiple sandbox backends (Daytona, Modal, E2B). (2) **Large-scale evaluation**: 8 models (Gemini-3.1-Pro/3-Flash, Opus 4.6/Sonnet 4.6/Haiku 4.5, GPT-5.4/5-mini/5-nano) × {Terminus-2 + one native harness} = 16 configurations over 54 benchmarks and 6,627 tasks, 3 trials each (~0.3M trajectories, 226B tokens, >$300K). Headline findings: model beats harness; benchmark rankings are highly correlated; frontier models use fewer tokens but cost more; unresolved tasks often reflect task defects rather than difficulty. (3) **Harbor-Index**: 82 audited tasks from 29 benchmarks; no configuration exceeds 30% pass, best GPT-5.5+Codex at 28.0%. The claims are mostly well-scoped; the Limitations section (§6, App. I) concedes the findings do not span all models, harnesses, or settings.
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 5 strengths & 5 weaknesses, analyzed in detail
- 6 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What is Harbor-Index and why was it created?
Harbor-Index is a curated meta-dataset of 82 tasks drawn from 29 agentic benchmarks, built by filtering for difficulty, auditing with AI and human reviewers, and iteratively fixing broken tasks. It exists because full agentic evaluations are expensive and many hard-looking tasks are actually defective; on this suite no model-harness configuration exceeds 30% pass rate, with the best (GPT-5.5 + Codex) at 28.0%, so a full run is affordable while leaving clear headroom for future models.
Does the choice of agent harness (scaffold) matter, or is it all about the model?
According to this paper, the base model matters far more: in a mixed-effects analysis, model fixed effects span 0.451 versus 0.087 for harnesses (about 5.2× wider). Harnesses still matter operationally — vendor-native harnesses cut matched-task timeout rates from 42% to 26% versus Terminus-2, partly because Terminus-2 only offers a bash action space — but they do not substitute for model capability.
How many benchmarks do you need to compare AI agents?
The paper's redundancy analysis finds the 54-benchmark matrix is low-rank: after greedily selecting 12 benchmarks, every remaining benchmark correlates at ρ≥0.7 with a selected one, and within a benchmark, just 3 tasks already recover the full system ranking at mean ρ≈0.923. It recommends prioritizing coverage of distinct capability axes over raw benchmark count.
Are the code, data, and trajectories from this paper publicly available?
Yes. The authors release the benchmark adapters (github.com/harbor-framework/harbor), Harbor-Index 1.0 (github.com/harbor-framework/harbor-index), the full evaluation trajectories and results, and an interactive results site at harbor-index.org.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- The provided text is a sampled excerpt with explicit gaps: one gap falls mid-Section 4 (between the Harbor-Index benchmark list and Figure 6), so Harbor-Index construction details (App. H), Table 1 values, and Figure 5 content were not visible; judgments about that section rely on the surrounding visible text and the abstract.
- A second gap truncates the reference list and parts of Appendices D.6.6-G.2.2 (per-benchmark details for Data & Analytics, Professional Domains, Safety, and Multimodal domains; full failure-mode results), leaving those areas unreviewed.
- Figure 4's failure-mode prevalences appear in the extraction only as axis/category text without numeric bar labels, so per-rubric percentages could not be verified; only the qualitative dominance claim and κ annotations are checkable.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.