Explaining AI Agents Through Execution Traces
Vittoria Vineis, Fabiano Veglianti, Lorenzo Antonelli, Claudia Di Carlo, Matteo Silvestri, Gabriele Tolomei
cs.AI
Quick verdict
Trace-grounded XAI pipeline for LLM agents beats naive LLM explanation (NFI 0.816 vs 0.061); rigorous stats but one annotator, one model, QA-only evaluation.
Summary
The paper introduces a post-hoc explainability framework that converts an LLM agent's raw execution trace into two artifacts: a structured, requirement-indexed report and a constrained natural-language narrative. The pipeline normalizes traces, decomposes the user goal into atomic requirements, and reconstructs per-step knowledge-action states that separately track behavioral commitments and internal/external evidence; deterministic validation forces every positive status to cite a trace pointer, and an eight-configuration taxonomy (Table 1) labels behavior-evidence concordance or mismatch. Evaluation covers 548 traces: GAIA (via TRAIL questions) and HotpotQA with ReAct and plan-and-act agents (gemini-3.1-flash-lite) plus external TRAIL traces. Against a single-prompt LLM baseline, the structured report reaches NFI 0.816 versus 0.061 on 63 human-referenced reports; the full pipeline's narratives beat direct LLM explanation on faithfulness, exhaustiveness, linguistic quality, and overall quality (LLM-judge M4 delta=0.62, exact permutation p<0.001; the human subset concurs). Statistical practice is unusually careful (query-level exact permutation, BH correction, cluster bootstrap, judge-calibration analysis), and the authors candidly report judge miscalibration. Limitations: ground-truth references come from a single annotator with no inter-annotator reliability; one model fills every LLM role; both benchmarks are QA; baselines are naive prompting variants; code and data are only 'available upon request'.
Review complete
Assessment at a glance
5
risk flags identified
● high 1
● medium 2
● low 2
5 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- A pipeline operating solely on execution traces produces structured reports substantially more faithful to human references than direct LLM prompting (NFI 0.816 vs 0.061 across 63 reports; Table 4).
- Narratives grounded in the structured report outperform direct LLM trace explanations on faithfulness, exhaustiveness, linguistic quality and overall quality for both human and LLM-judge evaluation (Tables 6 and 8).
- The structured report can serve as a practical proxy for trace-level faithfulness assessment: the judge flags 14/16 narratives that humans judge unfaithful via the M1-a_our cross-level comparison (Table 5).
- The advantages are direction-consistent across three agent architectures (ReAct, plan-and-act, TRAIL agent): none of 27 architecture-specific effects reverses sign (Tables 7 and 9).
Strengths
Auditable grounding contract, not just prompting
Every satisfied status requires supp(x) != empty (Section 3.1); deterministic validation discards ill-typed pointers and reconciliation to (Sigma_t, delta_t, Gamma_t) is model-free, yielding the 8-label diagnostic space of Table 1 (4 concordant incl. Genuine Gap; 4 mismatch types).
Unusually rigorous statistics for a small human study
Pre-specified hypotheses with M4 as confirmatory outcome, exact one-sided permutation at the query level (12 clusters, 2^12 enumerated), BH correction within hypothesis, cluster bootstrap B=5,000 explicitly flagged as indicative at k~12 (Section 4.2, Tables 6 and 8).
Candid LLM-judge validation including negative results
Clean ablation factorization
Reliability accounting of generation failures
Weaknesses
Ground truth from a single annotator
One LLM in every role
QA-only corpus, uneven per-source results
Baselines limited to prompting variants
No code or data release
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
This method paper (Sapienza University of Rome; arXiv 2609.06063, cs.AI) targets end-user explainability for tool-using LLM agents. The framework converts a raw execution trace into (i) a structured report X(tau) recording, per step, actions, observations, requirement-indexed knowledge-action states Sigma_t, transitions delta_t, and grounding reports Gamma_t; and (ii) a natural-language narrative rendered by a constrained narrator psi. Crucially (Section 3.1, Objective), the framework claims process transparency, not answer-correctness assessment or failure detection per se. Headline claims: trace-only portability across architectures; NFI 0.816 vs 0.061 over direct LLM report generation; narrative superiority on four criteria; proxy validity of the structured report for trace-level faithfulness.
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 5 strengths & 5 weaknesses, analyzed in detail
- 5 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What does 'Explaining AI Agents Through Execution Traces' propose?
A post-hoc explainability framework that takes the raw execution trace of a tool-using LLM agent and produces two artifacts: a structured report that indexes the agent's progress against decomposed user requirements (tracking behavioral commitments vs internal/external evidence, with mandatory citations to trace elements), and a constrained natural-language narrative generated only from that report. It explains the process, not the correctness of the final answer.
How well does the framework perform compared to just asking an LLM to explain the trace?
On 63 traces with human-annotated references, its structured reports score NFI 0.816 versus 0.061 for a single-prompt LLM baseline (74% of baseline reports score zero). Narratives from the full pipeline beat direct LLM explanation on faithfulness, exhaustiveness, linguistic quality and overall quality, for both human raters and an LLM judge (e.g., overall quality delta = 0.62 on the judge corpus, exact permutation p < 0.001).
What are the main limitations of the paper?
Ground-truth references were produced by a single annotator with no inter-annotator reliability reported; the same model (gemini-3.1-flash-lite) plays every LLM role including the baselines; evaluation covers only QA benchmarks (GAIA, HotpotQA); baselines are naive prompting variants rather than competing trace-analysis systems; and code/data are only 'available upon request' in this preprint.
Is the code or dataset available?
Not in the preprint: a title-page footnote states the source code and trace dataset are available upon request. The GAIA questions come from the public TRAIL corpus, but the authors' own ReAct and plan-and-act traces and their human annotations are not released.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Figures 1-3 are rendered graphics; their content was used only via captions and the in-text numeric descriptions (e.g., rating distributions and Spearman coefficients), not from the images themselves.
- Model names 'gemini-3.1-flash-lite' and 'Muse-Spark-1.2-Contributor' are quoted as printed in the extracted text; their real-world availability was not verified (no network access).
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.