Salesforce Koa: An Enterprise Language Model for Agentic Tool Use
Zixiang Chen, Sufeng Niu, Yingchi Liu, Wenting Zhao, Akshara Prabhakar, Shubham Mehrotra, Bin Bi, Zhujun Lan, et al.
cs.CL · cs.AI · cs.LG
Quick verdict
Spec-driven GRPO lifts an RL-post-trained base on agentic tool use (BFCL multi-turn 54.1→59.5), but gains over base are modest and lack significance testing.
Summary
Salesforce Koa is an enterprise language model created by GRPO post-training of the open-weight Nemotron-3-Super-120B base, using only public and synthetic data. Its distinctive pipeline compiles declarative Agent Script specifications into persona-conditioned multi-turn simulation environments run in NeMo Gym, where a frozen helper model simulates the customer, emulates tools, and judges task resolution; the reward is sub-question coverage gated by a duplicate-call filter, or binary state-equivalence in a deterministic sandbox. On Tau2Bench, BFCL, and a CRM benchmark, Koa modestly improves over its base (Tau2Bench weighted 68.64→69.41; BFCL 64.73→66.63%, with multi-turn rising 54.12→59.50%; CRM 0.84→0.86), surpasses GPT-4.1, and stays below Claude Opus 4.8 and GPT-5.5. Appendices report that SFT from the same already-RL-post-trained base helps single-turn tasks but not multi-turn tool use, and that reward stability depends on model scale. Main limitations: gains over the base are small and reported without variance or significance testing, the central spec-driven construction lacks an equal-budget ablation against generic synthetic tasks, the judge-based reward could be gameable, and no model, code, or data release is stated.
Review complete
Assessment at a glance
5
risk flags identified
● high 0
● medium 3
● low 2
5 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- Specification-driven RL, in which Agent Script agent definitions both configure the agent and supply simulated tasks plus grounded task-resolution rewards, improves agentic tool use of the Nemotron-120B base, most clearly on multi-turn benchmarks (BFCL multi-turn 54.12→59.50).
- Salesforce Koa surpasses GPT-4.1 on all three evaluated suites (Tau2Bench weighted 69.41 vs 54.48; BFCL 66.63 vs 53.96; CRM 0.86 vs 0.81) while remaining below Claude Opus 4.8 and GPT-5.5.
- From an already RL-post-trained base, RL improves multi-turn tool use substantially while SFT adds little (scoped observation, Appendix J).
- Reward-design conclusions do not transfer across scales: the gated-coverage reward trains stably on the 120B policy but collapses on the Nano-30B proxy after roughly step 30 (Appendix I).
Strengths
Authoring-to-training bridge is genuinely novel
Section 2.1 shows the same Agent Script specification that configures an Agentforce agent is statically compiled into a typed workflow graph that instantiates the RL environment and structures the reward criteria — a clean, reusable link between agent development and model post-training.
Transparent reward provenance and rollout anatomy
Figure 1 and Appendix B explicitly mark frozen vs trained components (policy only), the per-turn interaction order, and where the scalar reward originates; Eq. 1-2 define coverage and the duplicate-call gate precisely.
Useful ablations including negative results
Evaluation hygiene
Honest claim scoping
Weaknesses
Small gains over base with no statistical treatment
Central contribution lacks a controlled ablation
LLM-judge reward is potentially gameable
Weak baseline framing
Reproducibility not supported
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
This industrial technical report presents Salesforce Koa, a ~120B enterprise agent model produced by GRPO post-training of the open-weight Nemotron-3-Super-120B base using only public and synthetic data. Four claims are advanced: (1) specification-driven RL — compiling Agent Script agent definitions into persona-conditioned multi-turn simulation environments with grounded task-resolution rewards — improves agentic tool use; (2) Koa improves over its base on Tau2Bench, BFCL, and a CRM benchmark and surpasses GPT-4.1 while staying below Claude Opus 4.8 and GPT-5.5; (3) from an already RL-post-trained base, RL adds far more on multi-turn tool use than SFT (explicitly scoped, Sections 1 and 4, Appendix J); (4) reward stability is scale-dependent (Appendix I).
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 5 strengths & 5 weaknesses, analyzed in detail
- 5 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What is Salesforce Koa and what model is it built on?
Salesforce Koa is an enterprise language model for agentic tool use, created by Salesforce by reinforcing-learning fine-tuning (GRPO) the open-weight Nemotron-3-Super-120B model from NVIDIA. It was trained only on public and synthetically generated data — the paper states no customer data was used — and is specialized for multi-turn tool calling in business workflows such as CRM customer support.
How does Salesforce Koa perform compared to GPT-4.1 and frontier models?
Per the paper's Table 1, Koa beats GPT-4.1 on all three evaluated benchmarks: Tau2Bench weighted average 69.41 vs 54.48, BFCL 66.63% vs 53.96%, and CRM Bench 0.86 vs 0.81. However, it remains below the strongest frontier models tested — Claude Opus 4.8 (74.00 / 78.18 / 0.87) and OpenAI GPT-5.5 (83.99 / 67.63 / 0.90). Its clearest gain over its own base model is on multi-turn tool use (BFCL multi-turn 54.12→59.50).
What is the simulation-to-reward pipeline in the Koa paper?
It is the training-data engine: declarative agent specifications (Agent Script) are compiled into simulated multi-turn customer-service environments in NeMo Gym. A frozen helper LLM plays three roles in each rollout — simulated customer, tool emulator, and coverage judge. The reward equals the fraction of numbered task sub-questions the judge marks resolved (requiring tool-grounded answers for data-dependent questions), set to zero if the policy repeats identical tool calls; a deterministic sandbox variant instead scores exact final-state equivalence.
Is Salesforce Koa open-source, and does it use customer data?
The paper states training used only public and synthetic data with no customer data. It does not announce an open release of the model, code, or training data, so Koa should currently be treated as an internal Salesforce model described in a technical report.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Figures 1-4 are described in the extracted text but not visually verifiable; trend claims (e.g., reward collapse at step ~30 on Nano) rest on the authors' descriptions.
- This is a v1 arXiv preprint (2609.15066) and has not undergone peer review.
- Helper-model prompts and the coverage-judge rubric are only sketched in the paper; judgment of the reward's gameability is therefore indirect.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.