Harness Engineering in LLM Tool Use via Agent-Native Reusable Tool Primitives
Haibo Jin, Suijin Wang, Xucheng Yu, Haojing Luo, Haohan Wang
cs.SE · cs.AI · cs.CL · cs.LG · cs.MA
Quick verdict
HEART wraps tools in LLM interfaces plus planner/router/verifier; claims broad wins over frontier models, but no code release, no variance, uneven baselines.
Summary
This method paper introduces HEART, a framework for LLM tool use built on two components: Tool Primitives, which wrap each tool in an LLM interface so models invoke tools through natural language while schema resolution and execution are handled internally, and ToolFace, a centralized repository of 25,519 schema-function pairs from which relevant tools are retrieved at inference time rather than enumerated in context. A Planner-Router-Verifier multi-agent pipeline (all roles on Qwen3-8B, re-planning budget 3) orchestrates planning, parameter binding, execution, and feedback-driven recovery. Across five benchmarks (ToolBench, NESTFUL, τ2-Bench, ACEBench, BFCLv4), the authors report average Pass/Win Rate of 75.1%/75.7% on ToolBench, NESTFUL full-sequence accuracy of 0.44 (vs 0.40 for GPT-5.4), best Pass^k on all τ2-Bench domains (Telecom Pass4 0.50 vs 0.33 for Claude-4.6-Sonnet), ACEBench overall 86.9% vs 86.0% for GPT-5.4, and top BFCLv4 scores — claiming +10% over SFT models and +6% over three frontier commercial models at up to 85% lower API cost. On 50 self-curated real-world tasks, HEART reaches 84% completion versus a 22% commercial average, and shows 0.0% prompt-injection success because tool schemas never enter the prompt. Main limitations: no code or ToolFace release, no variance or significance reporting despite three runs, a small human evaluation (10 tasks/domain) whose setup gives HEART exclusive access to 54 hand-crafted tools, and minor prose-table inconsistencies.
Review complete
Assessment at a glance
5
risk flags identified
● high 2
● medium 3
● low 0
4 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- Wrapping tools in LLM interfaces (Tool Primitives) enables nested and multi-turn tool calling without exposing raw API schemas to the calling model.
- HEART with a Qwen3-8B backbone outperforms SFT-based tool-calling models by ~10% and GPT-5.4, Claude-4.6-Sonnet, and Gemini-3.1-Pro by ~6% on average across five benchmarks.
- Storing tool schemas in ToolFace and invoking via Tool Primitives eliminates the ToolHijacker prompt-injection attack surface, yielding 0.0% attack success rate on ToolBench.
- On 50 self-curated real-world tasks HEART completes 84% versus a 22% average for three frontier commercial models, at up to 85% lower API cost.
Strengths
Well-motivated abstraction grounded in documented failure modes
Sec. 1 anchors the design in measured failures: 28% full-sequence accuracy for the strongest models on NESTFUL and 7%-85% accuracy drops as tool catalogues scale from 8K to 120K tokens, making the case for natural-language tool interfaces and on-demand retrieval concrete rather than speculative.
Consistent multi-benchmark evaluation with transparent overhead accounting
One fixed configuration (Qwen3-8B for all four roles, B=3, three runs) is evaluated on ToolBench, NESTFUL, τ2-Bench, ACEBench and BFCLv4; Table 3 honestly reports HEART's ~3x token consumption (e.g., 21,684 vs 7,153 tokens Retail) and higher latency alongside its cost advantage.
Informative ablation and design-space studies
Unusually complete specification for reproduction
Weaknesses
Baseline fairness confound in the headline real-world study
No dispersion or significance testing anywhere
Internal number inconsistencies
Self-evaluation circularity
No code, harness, or ToolFace release
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
The paper (arXiv:2609.01736) proposes HEART, a training-free harness for LLM tool use built on two artifacts: **Tool Primitives**, LLM-wrapped interfaces that accept natural-language requests and internally resolve API schemas and execute functions (Sec. 3.2, Eq. 1), and **ToolFace**, a registry of 25,519 manually authored schema-function pairs aggregated from ToolBench (16,464), NESTFUL (4,348 coding + 40 math), ACEBench (4,538), τ2-Bench (68), BFCLv4 (7), and 54 hand-crafted tools — the arithmetic checks out exactly. A Planner-Router-Verifier pipeline (Sec. 3.3, Algorithm 1) adds intent/sufficiency checking, parameter binding, four-criterion verification, and feedback-driven re-planning with budget B=3. Central claims: +10% over SFT models and +6% over GPT-5.4/Claude-4.6-Sonnet/Gemini-3.1-Pro across five benchmarks, up to 85% lower API cost, 84% vs 22% task completion on 50 self-built real-world tasks, and 0.0% prompt-injection ASR.
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 4 strengths & 5 weaknesses, analyzed in detail
- 5 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What is HEART in LLM tool use?
HEART (Harness Engineering via Agent-native Reusable Tool Primitives) is a training-free framework in which a Planner decomposes the user request, a Router binds parameters, LLM-wrapped Tool Primitives execute tools, and a Verifier checks results and triggers re-planning on failure. It uses Qwen3-8B for all roles and retrieves tools on demand from a 25,519-function repository called ToolFace instead of putting whole tool catalogs into the prompt.
What is a Tool Primitive?
A Tool Primitive is a single tool wrapped in an LLM interface: it receives a natural-language request, internally maps it to the tool's API schema (type coercion, constraint validation), executes the function, and returns a structured result. The calling model never sees raw API schemas, which the authors argue enables nested multi-step calls and removes the prompt-injection surface at tool-selection time.
Does HEART really beat GPT-5.4 and Claude?
The paper reports that an 8B-backbone HEART outperforms GPT-5.4, Claude-4.6-Sonnet, and Gemini-3.1-Pro by about 6% on average across five tool-use benchmarks (e.g., ACEBench overall 86.9% vs 86.0%), and 84% vs ~22% on its own 50 real-world tasks. These are the authors' numbers without released code or significance testing, and the real-world setup gives HEART exclusive access to 54 hand-crafted tools, so independent verification is still pending.
Is HEART's code or the ToolFace repository available?
The paper does not mention any code, data, or ToolFace release. The 25,519 tool schemas are described as manually authored, and no repository link appears in the text, so reproduction currently requires re-authoring the registry and evaluation harness from scratch.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- The metadata abstract is cut mid-sentence (ending at 'Gemini-3.1-P'), but the complete abstract is present in the extracted full text, so no content was lost.
- Figure 1 is extracted only as scattered label text, so the diagram itself could not be inspected; all quantitative claims were checked against the tables, which extracted legibly.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.