Is Bash All You Need? An Empirical Study of Tool Interfaces for Enterprise Digital Worker Agents
Hazel Mak, Susheel Suresh, Sahil Bhatnagar, Barry Wang, Chhaya Methani, Alejandro Gutierrez Munoz
cs.SE · cs.AI · cs.CL
Quick verdict
Claims bash-only beats typed tools and PTC on enterprise agent tasks; controlled ablation, but the self-built tool catalog may weaken the Tool-only baseline.
Summary
This Microsoft study asks whether a plain Unix shell outperforms curated typed-tool catalogs for enterprise LLM agents. The authors ablate five interfaces — typed tools only, tools plus bash, bash alone, bash with persistent agent-synthesized tools, and programmatic tool calling (PTC) — on TheAgentCompany (174 tasks, a 60-tool author-built catalog) and APEX-Agents (480 tasks, 20 MCP tools), each with Opus-4.8 and GPT-5.5 under identical base prompts and stopping criteria. Bash alone posts the highest score in every benchmark-model combination (roughly +4.8–7.4 pp on APEX-Agents and +21.8–24.5 pp on TheAgentCompany) while using 19–72% fewer total tokens. Adding tools or synthesized-tool libraries to bash shows no detectable pooled score gain (paired bootstrap CIs include zero), and PTC saves tokens over direct typed calls but scores below bash. Limitations: the TheAgentCompany catalog is author-constructed, so Tool-only underperformance may partly reflect catalog design; APEX scoring relies on an LLM judge with no reported human agreement; each arm ran once, so bootstrap CIs capture task sampling but not run-to-run variance; and internal benchmark patches limit comparability with published scores.
Review complete
Assessment at a glance
5
risk flags identified
● high 0
● medium 4
● low 1
5 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- Bash alone outscores typed-tool-only interfaces in every benchmark-model arm (+21.8–24.5 pp on TheAgentCompany, +4.8–7.4 pp on APEX-Agents) while using 19–72% fewer total tokens.
- Adding a typed catalog or persistent agent-synthesized tools to a bash agent produces no detectable pooled score gain in paired comparisons.
- PTC reduces uncached input tokens versus direct typed calls (e.g., −97.6k tokens/task on TheAgentCompany, Table 6) with broadly similar scores, but underperforms bash alone in both quality and cost efficiency.
- PTC has the lowest tool-call success rate of all interfaces (65.2% on TheAgentCompany, Table 2), attributed to the difficulty of coordinating calls inside generated programs.
- Practitioners should prefer bash-only agents where arbitrary execution can be isolated, and PTC where compliance requires restricting actions to an approved catalog.
Strengths
Cleanly controlled multi-arm ablation
Five interfaces × two benchmarks × two models (20 arms) share the same task text, base prompt, and stopping criteria (§3, §4); paired task-model analysis (Table 9, N=348/960) isolates interface effects from task difficulty.
Candid confound management
Appendix B.1 documents observed reward hacking (agents reading evaluator scripts from the public repo, reading simulated-coworker files) and applies uniform mitigations — network blocking and encrypted coworker info — across all interfaces rather than hiding the problem.
Mechanism telemetry beyond headline scores
Practitioner-relevant negative results
Scoped, deployment-aware recommendations
Weaknesses
Author-built catalog confounds the Tool-only baseline
LLM judge on APEX-Agents without human validation
Single-run arms understate uncertainty
PTC is a custom runtime, not the cited vendor APIs
Weak comparability of absolute scores
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
The paper asks whether a general shell outperforms curated typed-tool catalogs for enterprise LLM agents. It runs a 5×2×2 ablation — five interfaces (Tool-only, Bash+Tool, Bash, Bash+Synthesis, PTC) on TheAgentCompany (174 tasks) and APEX-Agents (480 tasks), with Opus-4.8 and GPT-5.5 — yielding 20 arms under shared base prompts and stopping criteria (§3, §4). Three central claims: (1) bash alone beats typed tools in every arm tested (+21.8–24.5 pp on TheAgentCompany, +4.8–7.4 pp on APEX-Agents) at 19–72% fewer total tokens; (2) adding tools or persistent synthesized tools to bash yields no detectable pooled gain; (3) PTC trades quality for tokens relative to bash but is cheaper than Tool-only. Claims are explicitly scoped to current frontier models, with the authors warning results may not carry to later releases.
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 5 strengths & 5 weaknesses, analyzed in detail
- 5 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What is the main finding of 'Is Bash All You Need?'
That agents given only a bash shell beat agents using typed tool catalogs or programmatic tool calling (PTC) on two enterprise benchmarks (TheAgentCompany and APEX-Agents) with both Opus-4.8 and GPT-5.5 — scoring roughly 4.8–24.5 percentage points higher while using 19–72% fewer tokens — and that adding tools to a bash agent did not measurably help.
When should I use PTC instead of a bash-only agent?
Per the authors: use bash-only when you can safely isolate arbitrary code execution (it scored best and cheapest); use PTC when security or compliance rules require the agent's actions to stay inside an approved tool catalog. PTC also cuts token use versus direct one-at-a-time tool calls.
Does giving a bash agent extra tools improve its performance?
In this study, no. Paired comparisons found no detectable pooled score gain from adding typed tools (Bash+Tool − Bash: −0.6 pp on TheAgentCompany, −0.2 pp on APEX-Agents) or persistent synthesized tools (−1.6 and −0.2 pp), with confidence intervals including zero in all cases.
Are the paper's scores comparable to published TheAgentCompany or APEX-Agents leaderboard numbers?
No. The authors patched simulated-cowower reliability bugs, built their own 60-tool catalog, used a June 2026 APEX snapshot with their own LLM judge, and ran default rather than maximal reasoning effort — so absolute scores are not directly comparable to public leaderboard results.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Figure-embedded values (Figure 4 per-bin means, Figure 5/6 bar heights) are not present as numbers in the extracted text; assessments of those figures rest on the surrounding prose only.
- The actual system-prompt texts are absent from the paper itself (Appendix A describes only their structure), so prompt-confound assessment is limited to what the text states.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.