CliffCompaction: Cost-Efficient Compaction for Long-Horizon Coding Agents
Trang Nguyen, Eulrang Cho, Bingqing Chen, Tim Dettmers, Tim Dettmers
cs.AI · cs.LG · cs.SE
Quick verdict
Truncation-only compaction cuts coding-agent cost ~50% at maintained performance with strong KernelBench results; stats testing and fair baselines are thin.
Summary
CliffCompaction is a training-free, rule-based context-management method for long-horizon coding agents. Instead of LLM summarization it only truncates or drops content: tool results over 500 characters are discarded, tool calls reduced to ~150-character signatures, thoughts cut to 300 characters, while the system prompt, task description, and two recent turn pairs stay verbatim. When context crosses a token threshold it compacts once and discards the previous compacted block, so no summary-of-summary drift accumulates; only implicit residual knowledge carries forward. On SWE-bench Verified (mini-swe-agent), success stays within roughly 2 points of full context at 16K thresholds for most models; on Terminal-Bench 2.0 (89 tasks) it exceeds the full-context baseline (61.42% vs 59.16% for Kimi K2.6 at 32K/16K) while roughly halving cost, driven by an 80% cut in cache-read spend (Table 7). Combined with a LightGBM rollout selector (SGV), three compacted Kimi K2.6 rollouts reach 69.7% for $58.01, matching Opus 4.7 (69.4%, system-card score) for less than one GPT-5.3 Codex run. On KernelBench Level 3 it reaches 2.23x/3.58x speedup at 200/400 steps across million-token sessions, above AdaExplore (1.78x) and CUDA-Agent (1.80x, different protocol). Limitations: no significance tests, seed counts unstated, savings collapse for some scaffold/model pairs (Table 8), and plain summarization nearly matches it on KernelBench (3.47x/$8.10 vs 3.58x/$8.32).
Review complete
Assessment at a glance
5
risk flags identified
● high 0
● medium 3
● low 2
5 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- Truncation-only, non-recursive compaction preserves or improves coding-agent success at up to ~50% lower cost, e.g., Kimi K2.6 on Terminal-Bench 2.0 reaching 61.42% at $0.19 vs 59.16% at $0.40 full-context.
- Discarding each previous compaction (never compacting a compaction) prevents accumulating context drift and sustains continual-learning sessions beyond one million tokens.
- With the SGV selector, three 16K-compacted Kimi K2.6 rollouts reach 69.7% on Terminal-Bench 2.0 for $58.01, matching Opus 4.7 (69.4%) and beating GPT-5.3 Codex (64.7%) for less than one Codex run.
- On KernelBench Level 3, a generic OpenHands agent with CliffCompaction surpasses purpose-built kernel-optimization systems, reaching 2.23x/3.58x at 200/400 steps vs AdaExplore 1.78x and CUDA-Agent 1.80x.
Strengths
Deployable, training-free design with no auxiliary LLM calls
Algorithm 1 is ~15 lines of deterministic rules (drop tool results >500 chars, signatures <=150 chars, thoughts <=300 chars, keep 2 recent turn pairs); the scaffold-agnostic API proxy is open-sourced and validated against closed-source Claude Code (Table 2: 76.69% vs 70.97% at matched ~45K on Terminal-Bench 2.1).
Unusually careful cost accounting
Costs are recomputed with a deterministic perfect-cache model from prompt lengths, then cross-checked against provider-metered real costs (Table 8); Table 7 decomposes the win: cache-read spend -80% ($0.427 to $0.087) against only +$0.06 re-prefill/output overhead, netting -51% total.
Breadth across benchmarks, scaffolds, and models
Mechanistic support for the precision-over-recall hypothesis
Honest reporting of failure modes
Weaknesses
No statistical inference anywhere
Mixed-provenance proprietary baselines behind the headline claim
The headline scaling result depends on a supervised selector trained in-distribution
KernelBench comparison has protocol asymmetries the framing understates
Core hyperparameters are never ablated
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
CliffCompaction (arXiv 2609.26779) is a method paper on context management for long-horizon coding agents, backed by broad empirical evaluation: SWE-bench Verified, Terminal-Bench 2.0/2.1, and KernelBench Level 3, across mini-swe-agent, OpenHands, Terminus-2, and Claude Code (via an API proxy), with roughly ten Kimi (K2.5-K2.7) and GLM (5-5.3 Flash) configurations plus proprietary baselines. Four central claims: (1) rule-based, truncation-only compaction preserves or improves success at up to ~50% lower cost; (2) discarding prior compactions avoids summary-of-summary drift; (3) with a learned selector (SGV), three compacted Kimi K2.6 rollouts match Opus 4.7 (69.7% vs 69.4%) for $58.01; (4) on KernelBench L3 the approach beats purpose-built kernel-optimization systems (3.58x at 400 steps vs AdaExplore 1.78x, CUDA-Agent 1.80x). The framing is ambitious, but section 8 honestly concedes the benefits are scaffold- and horizon-dependent.
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 5 strengths & 5 weaknesses, analyzed in detail
- 5 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What is CliffCompaction?
A rule-based context-compaction technique for long-running coding agents. Instead of asking an LLM to summarize conversation history, it only truncates or drops content: long tool outputs are removed, tool calls are reduced to short signatures, and when a token threshold is hit, the new compaction replaces rather than builds on the previous one, preventing drift from chained summaries. It runs as an API proxy and requires no training.
How much cost does CliffCompaction save?
It depends heavily on scaffold and model. Best case is roughly 50% per task: on Terminal-Bench 2.0, Kimi K2.6 drops from $0.40 to $0.19 per instance and GLM 5.1's total cost falls 51%, with cache-read spending down 80%. On SWE-bench Verified with mini-swe-agent, savings for Kimi K2.6 are near zero because those trajectories are already cache-efficient.
Does it beat Claude Code's built-in compaction?
In the paper's Terminal-Bench 2.1 test with GLM 5.3 Flash at a matched ~45K peak context, CliffCompaction scored 76.69% versus 70.97% for Claude Code's native auto-compaction (summarization-based); both beat the default 200K configuration at 73.03%.
Is the code available?
Yes. The authors open-source a scaffold-agnostic API-proxy implementation that works with Claude Code, Codex, and other harnesses (github.com/nguyenvuthientrang/cliffcompaction).
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Figures 1-7 are available only as caption text; plotted data could not be independently verified, notably Figure 7b, whose extracted endpoint values appear to order baselines opposite to the text's claim.
- The number of evaluation seeds is not stated in the extracted text, so +/- values cannot be attributed to seed vs task variance.
- Terminal-Bench leaderboard entries and the Opus 4.7 system-card score cited in Table 4 could not be checked offline; the review takes them at face value.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.