Multi-Step Tool-Calling over Korean Open Public APIs: A Benchmark and a Data-Synthesis Recipe
Dain Kim, Eungi Cho, Kyumin Kim, Shinyeong Noh, Kyuseong Lim
cs.AI · cs.CL
Quick verdict
KOPA-Bench/EDGE deliver a credible live-API benchmark and execution-grounded synthesis with solid fine-tuning gains, though ablations are self-referential and the headline 9B-vs-27B claim is untested.
Summary
The paper introduces KOPA-Bench, 145 multi-step tool-calling tasks over ten live Korean public-API platforms, where chains of dependent calls (API-key retrieval, pagination and aggregation of high-cardinality responses) are required, and EDGE, a data-synthesis pipeline that builds a tool dependency graph, prunes links that fail under live execution, and types junctions by response cardinality to convert many-record outputs into bounded fan-out or deterministic reduction, yielding 1,781 execution-verified training trajectories. GRPO fine-tuning lifts Qwen3.5-4B and 9B pass@1 by 13 and 10 points respectively; the 9B model nearly matches its untuned 27B sibling and also improves on the out-of-distribution BFCL benchmark. Gains also hold on the 31 tasks built from platforms entirely withheld from training. Key limitations: Korean public-API specificity, live endpoint drift complicating exact reproduction, and ablations that compare only the method's own variants without end-to-end comparison against competing synthesis systems.
Review complete
Assessment at a glance
4
risk flags identified
● high 0
● medium 3
● low 1
4 strengths · 4 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- KOPA-Bench is the first multi-step function-calling benchmark over live Korean public APIs, requiring dependent call chains and high-cardinality response handling.
- EDGE synthesizes executable multi-step trajectories by verifying LLM-proposed tool dependencies against live APIs and resolving many-record handoffs via cardinality typing.
- GRPO fine-tuning on the 1,781-task EDGE corpus raises Qwen3.5-4B/9B pass@1 on KOPA-Bench by +13pp/+10pp, with the 9B model nearly matching the untuned 27B sibling.
- Gains generalize out-of-distribution to BFCL and to 31 tasks on platforms entirely withheld from training data.
- The contamination audit shows 0% verbatim query leakage and zero dependency-edge overlap despite 62.2% tool-universe overlap.
Strengths
Genuinely live, multi-axis benchmark
KOPA-Bench (§2) spans 145 tasks over 10 platforms and 6 domains backed by 2,318 live functions via MCP servers, with RESPONSE/ENVIRONMENT/ACTION evaluation and two-stage expert validation (audit found errors in 7/145 tasks, 4.8%).
Execution-grounding is validated, not just asserted
§4.3/Fig. 4: the LLM-judged skeleton executes 50.2% of the time vs 62.7% for the converged graph G*, and pruned edges only 14.8% (70.5% of pruned edges never succeed in any trial), showing the loop removes exactly the edges live APIs reject.
Unusually careful statistics and contamination auditing
Clean objective and filtering ablations
Weaknesses
No end-to-end external baseline
Headline 'nearly matches 27B' lacks a significance test
ACTION metric is confounded by solve rate
Validity tied to uncontrolled live endpoints
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
This LG CNS paper targets on-premise government agents: open-source LLMs must chain calls over live Korean public APIs, where two failure modes dominate — skipped prerequisite lookups (e.g., missing API-key retrieval) and premature answers from the first page of paginated, high-cardinality responses (Figure 1). It delivers (1) **KOPA-Bench**, 145 tasks over 10 platforms / 6 domains (Table 5), backed by 2,318 live functions behind MCP servers, and (2) **EDGE**, a synthesis pipeline that mines executable multi-step trajectories from live execution. Headline results: GRPO on 1,781 EDGE trajectories lifts Qwen3.5-4B pass@1 0.1758→0.3094 and Qwen3.5-9B 0.3275→0.4310 (Table 1), the 9B model approaching the untuned 27B (0.4482); BFCL multi-turn also improves (+5.87pp for 9B).
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 4 strengths & 4 weaknesses, analyzed in detail
- 4 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What is KOPA-Bench?
A benchmark of 145 multi-step tool-calling tasks built on ten live Korean public-API platforms across six domains (traffic, finance, education, law, politics, district administration). Each task requires chaining dependent calls — e.g., fetching an API key first — and handling responses that return many records, so the agent must paginate and aggregate instead of answering from the first page.
What does EDGE do?
EDGE is a data-synthesis pipeline for tool-calling training data. It builds a graph of how one API tool's output can feed another's input, keeps only links that succeed when actually executed against the live APIs (via Thompson-sampling edge probing and pruning), and then synthesizes verified multi-step trajectories, typing each junction by how many records flow through it so high-cardinality responses become bounded fan-out or deterministic reduction.
Can a 9B model really match a 27B model at function calling?
On KOPA-Bench, the GRPO-tuned Qwen3.5-9B scores pass@1 0.4310 versus 0.4482 for the untuned Qwen3.5-27B — close but not equal, and the paper reports no statistical test for this specific comparison. The tuning gains themselves (9B: 0.3275→0.4310) are supported by non-overlapping confidence intervals over eight seeds.
Is the code and data available?
Yes. The paper states that code and data are available at the public repository github.com/dneirfi/EDGE-KOPA, and the appendices release all prompt templates and hyperparameter values used for synthesis, filtering, and evaluation.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Review is based on the PDF-extracted text; figure content (e.g., Figures 3, 4, 6, 7) is summarized only via captions and in-text numbers since plotted detail is not extractable.
- Appendices D-J are present but some per-task metadata (e.g., the full platform tool list) is referenced to external material rather than enumerated in the text.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.