HEXIS: Compiling Skills into Extended Finite State Machines
cs.AI
Quick verdict
Compiles agent skills into FSMs, separating knowledge from control flow; consistent gains across 4 benchmarks, but small test sets lack significance testing.
Summary
HEXIS addresses a failure mode of LLM agents that follow written 'skill' documents: the model must repeatedly infer which operation comes next, so prescribed steps get skipped or misapplied. The system compiles a skill document plus development execution traces into an extended finite state machine in which states hold local prompts (skill knowledge) and typed variables, while guarded transitions enforce ordering, branching, loops, and termination. An incremental compiler first drafts a machine from the document with static validation, then aligns each development trace with existing states, adding or reusing states; an update is accepted only if static checks and replay of the current and all previously archived traces pass. Across SpreadsheetBench, LiveMath, DABench, and LongSeal with four executors, HEXIS beats native Skill+ReAct in 15 of 16 settings (average +16.1 points), transfers across backbones without recompilation, and cuts Qwen3.8-27B execution tokens by 38.4-88.9%; combining with SkillOpt reaches 84.2% on SpreadsheetBench. Main limitations: no statistical significance testing on test sets of 51-121 tasks, closest trace-compilation baselines not run, compilation cost unreported, gains on LongSeal are marginal, and compliance checks are authored by the authors themselves.
Review complete
Assessment at a glance
4
risk flags identified
● high 1
● medium 3
● low 0
5 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- Compiling skill documents into extended FSMs separates task knowledge (in-state prompts) from control flow (guarded transitions), reducing control deviations that compound over long horizons (§3.1, Eq. 3).
- Trace-guided incremental compilation accepts an update only when static checks and replay of the current and all previously archived traces pass, preserving every validated execution path (Eq. 9, App. B.3.4).
- HEXIS beats native Skill+ReAct in 15 of 16 benchmark-by-executor settings (+16.1pp average) and transfers across backbones without recompilation (Table 1).
- Qwen3.8-27B reduces execution tokens by 38.4-88.9% under HEXIS, and SkillOpt+HEXIS reaches 84.2% success at 69k tokens per task on SpreadsheetBench (Table 2).
Strengths
Well-motivated problem formalization
§3.1 derives cumulative deviation probability (Eq. 3) from per-step deviation εt, giving a crisp rationale for externalizing control; the Prompt-only ablation (Table 6) empirically confirms that merely describing the compiled control flow in text is insufficient (33.9% vs 76.7% on LiveMath), isolating programmatic enforcement as the active ingredient.
Conservative update discipline
Acceptance (Eq. 9, App. B.3.4) requires static checks plus replay of the current and every previously archived trace using recorded tool outputs; rejection leaves machine and archive untouched. App. H shows the frozen machine still yields 227/287 successes across LiveMath groups 21-49 without further updates.
Breadth and transfer of evaluation
Honest negative reporting on cost
Artifact transparency
Weaknesses
No statistical inference
Metric construction opacity and potential circularity
Closest baselines not run
Compilation cost unreported
Marginal LongSeal gains and per-benchmark machines
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
HEXIS (ICLR 2027 format, single author) targets procedural compliance in LLM agents that execute "skill" documents (§1): when the whole document is context, the model must repeatedly infer both the task reasoning and which operation comes next, and per-step deviation probabilities compound over horizons (Eq. 3). The proposed remedy compiles a skill document, plus development execution traces, into an extended finite state machine M = (Q, q0, V, a, E, F, τ) (Eq. 6) where model states carry local skill prompts and typed read/write sets, and ordered guarded transitions (Eq. 8) deterministically select the successor operation from stored variables. Claimed contributions: the compilation scheme, a trace-guided incremental compiler gated by static checks plus replay of all archived traces, and an evaluation showing +16.1pp average success over Skill + ReAct across 4 benchmarks x 4 executors, cross-backbone transfer without recompilation, and 38.4-88.9% token reductions for Qwen3.8-27B.
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 5 strengths & 5 weaknesses, analyzed in detail
- 4 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What does HEXIS do?
HEXIS compiles an agent skill document (instructions and domain knowledge written for LLM agents) into an extended finite state machine. States hold local prompts and typed variables; guarded transitions enforce step order, branching, loops, and termination, so the runtime rather than the model decides which operation comes next. Machines are built from the document plus development execution traces and refined incrementally with replay-based validation.
How much does HEXIS improve task success?
Across four benchmarks (SpreadsheetBench, LiveMath, DABench, LongSeal) and four executor models, HEXIS beats supplying the skill document to a ReAct agent in 15 of 16 settings, by 16.1 percentage points on average. Gains are largest on LiveMath (+31.4 to +38.0 points) and smallest on DABench (1.9-4.0 points), where native success was already 78-86%.
Is the HEXIS code available?
The paper states that code and experimental artifacts will be released at an anonymous repository (anonymous.4open.science/r/HEXIS-575F/). The benchmarks, skills, and baseline implementations used are public with licenses listed, but this review could not verify the repository's contents or completeness.
What are the main limitations of HEXIS?
The method depends on the quality and coverage of the skill document and development traces, so requirements never expressed there may be missing. Validation guarantees only replay of recorded paths, not correct reasoning within states or coverage of unseen situations. The reported evaluation also lacks statistical significance testing, and compilation cost is not reported.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Figures 1-5 and H.1 are extracted as raw text; visual details (e.g., per-axis token scales in Fig. 5) could not be independently verified.
- Figure 4's compliance values arrived as unformatted number runs; minor transcription artifacts are possible.
- The cited models (Claude Fable 5.1, qwen3.6-flash, Qwen3.5-9B, Qwen3.8-27B) and several 2026 preprints could not be verified offline.
- Code availability is a stated release promise at an anonymous repository; contents unverified.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.