Ask the Tool, Don't Guess: Agent Tool Calls Hold Their Progress, and the Serving System Should Read It
Yipeng Liu, Yingqiang Zhang, Feifei Li, Huanchen Zhang
cs.DC · cs.AI · cs.OS
Quick verdict
Proposes tools report live progress for KV-cache decisions; strong census, harness, and vLLM evidence, though timely-trigger rates (~17-25%) limit gains.
Summary
LLM agents spend much of a request's wall-clock time waiting for tool calls while their KV cache pins GPU memory. Serving systems decide whether to keep, evict, or restore that cache by predicting durations from tool names, histories, declared estimates, or engine occupancy — all fixed before the call runs. The authors show empirically that such estimates cannot even rank long calls, because durations are set by machine load and external services, and propose instead that running tools report progress through a side channel. A census of four public agent corpora finds strong (fraction-remaining) or weak (end-is-near) signals covering 66-72% of tool time. A harness recovers these signals — re-enabled progress bars, verbose switches with dry-run totals, disk observers, instrumented agent scripts — while keeping the agent's observations byte-identical, at under 1% wall-clock overhead and no significant benchmark-score change on 100 SWE-bench Verified tasks. At KV decision points the resulting remaining-time estimate is several times to an order of magnitude more accurate than reimplemented Continuum, CacheWise, TokenCake, and InferCept predictors, and stays accurate under load changes. Integrated into vLLM on 4x H100, it cuts p90 post-tool TTFT by 20.7% (HBM) and 20.8% (HBM+DRAM) versus LRU, near oracle. Main limitations: timely prefetch triggers land in only ~17-25% of runs, gains concentrate in long calls, and the anti-lying credit mechanism is simulation-only.
Review complete
Assessment at a glance
5
risk flags identified
● high 0
● medium 3
● low 2
5 strengths · 5 weaknesses · 5 author questions
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- No estimate fixed before a tool call starts can know or even rank its duration, because long-call durations are determined at run time by machine load, sandbox neighbors, and external services (Sec 2.1, Fig 3).
- Once revealed, most tool time — 66-72% across three corpora — carries a readable progress signal in two strengths: a fraction of work remaining, or an accurate end-is-near marker (Sec 2.2, Table 2).
- A harness can extract these signals without changing what the agent observes, at under 1% wall-clock overhead and no significant benchmark-score cost (Sec 3, Table 4).
- At KV-cache decision points, the reported progress is several times to an order of magnitude more accurate than the best published predictors and remains accurate when the environment's load changes (Sec 4, Figs 7/12).
- Plugged into vLLM via a few hints, progress-guided cache control cuts p90 post-tool TTFT by 20.7% (HBM) and 20.8% (HBM+DRAM) versus LRU, near oracle, while a credit-ledger mechanism bounds lying sessions (Sec 5).
Strengths
Well-motivated gap backed by quantitative evidence
Sec 2.1 quantifies both halves of the problem: fewer than 3% of calls hold ~2/3 of tool time (Fig 2), replayed commands run at median 0.12x recorded duration, and the rank correlation of long-call durations between recorded and idle-replay runs is Kendall tau = -0.04 (Fig 3c) — a crisp empirical basis for why any pre-call estimator fails, aptly echoing the database progress-estimator literature (Chaudhuri et al.).
First corpus-scale census of tool-call progress
Sec 2.2 (Table 2, Figs 4/11) gives a reusable taxonomy — two strengths, five kinds, four sources (A-D) — with per-corpus shares of tool time (strong 38.2-50.6%, weak raising coverage to 66-72%) across four public corpora plus stratified sampling and multi-model labeling; valuable independently of the system contribution.
Strict invariant discipline on the agent side
End-to-end evaluation on a production engine with oracle bounding and attribution
Treats the signal as untrusted input
Weaknesses
Modest absolute trigger timeliness, framed against near-zero baselines
Coverage, not precision, is the binding constraint
LLM-generated census labels without reported agreement statistics
Single-sandbox replay evaluation
Adversarial defense is a single preliminary simulation
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
- All 5 strengths & 5 weaknesses, analyzed in detail
- 5 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What problem does this paper solve?
LLM coding agents spend minutes waiting for tool calls (installs, builds, test suites) while their KV cache occupies scarce GPU memory. The serving system must decide whether that cache stays, is evicted, or is brought back in time, and today it guesses the tool's duration from its name, past calls, or a declared estimate. The paper shows such guesses fail for the long calls that matter most and proposes that running tools report their progress directly instead.
What are the main results?
Across four public agent corpora, recoverable progress signals cover 66-72% of tool time. The resulting remaining-time estimates are several times to an order of magnitude more accurate than prior predictors near a call's end, and stay accurate when machine load changes. Integrated into vLLM on 4x H100 GPUs, they cut the 90th-percentile time-to-first-token after tool calls by about 20.7-20.8% versus LRU, close to an oracle bound.
Does extracting progress change the agent's behavior?
By design, no. The harness routes progress over a separate side channel so the agent sees byte-identical tool output. On 100 SWE-bench Verified tasks, benchmark scores, steps, and token counts changed within bootstrap 95% confidence intervals, and the wrapper added under 1% to wall-clock time.
What if a tool lies about its progress?
Reports are treated as untrusted input. Each session earns credit when its reports prove correct and spends credit whenever the engine acts on a report, with separate ledgers for memory held and latency risked; a session out of credit falls back to default treatment. A preliminary simulation found lying raised a liar's held memory by only about 6%, of which the ledgers reclaimed roughly 5%.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Figures are available only through captions and numbers quoted in the text; curve shapes (e.g., the Fig 8 lead-time curves) could not be independently inspected.
- The per-phase decision-replay attribution (Sec 5.2, 63%/17% recovery) appears only in prose; any underlying table may be among the unextracted portions.
- No artifact or code-release statement was found in the extracted text; availability may be stated in parts not included in the extraction.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.