MIRAGE: How Conversation State Shapes Historical Evidence Use in Multimodal Personal Agents
Yu Liu, Wenxiao Zhang, Cheng Hu, Cong Cao, Fangfang Yuan, Xinyu Wang, Jin B. Hong, Yanbing Liu
cs.CL · cs.AI
Quick verdict
Controlled study showing pre-/post-compaction are distinct grounding-failure regimes; internally convincing, but six artifacts and one runtime bound generality.
Summary
MIRAGE is a controlled empirical study of whether multimodal personal agents genuinely ground answers in previously encountered evidence as conversation state changes. The protocol plants six multimodal evidence objects (ChartQA/ScreenSpot-derived) into a trunk conversation, appends seeded topic-disjoint filler turns, materializes checkpoints at four states (EIT ~23k, 50k, 80k, and ~100k after one native compaction), and issues 200 fixed probe questions (100 answerable/100 unanswerable) at each state on freshly restored checkpoints. Responses are scored deterministically along a grounding chain: answerability (AC), source attribution (SC), value correctness (VC), grounded correctness (GC=SC*VC), plus a mirage index (MI) and retrieval-path labels (context vs. tool). Across seven backbones on an OpenClaw runtime, GPT-5 stays stable (DS=3.4%) while open models collapse at depth (Qwen3-VL-4B retention 1.9%; MI up to 0.78); open models rarely switch to tool retrieval even when provenance fails (WS up to 100%); and mandatory artifact_recall improves deep pre-compaction SC/GC for tool-compliant Qwen models but regresses all of them after compaction. Limitations: a six-artifact corpus, a single runtime stack, retrieval pressure untested on frontier models, and confidence intervals covering only representative cells.
Review complete
Assessment at a glance
5
risk flags identified
● high 0
● medium 3
● low 2
5 strengths · 5 weaknesses · 5 author questions
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- Pre-compaction depth and post-compaction continuation form distinct, non-monotonic failure regimes rather than points on a single degradation curve (Delta-comp spans +0.33 to -0.05 across models).
- Outcome-only evaluation overestimates true evidence use: mirage rates reach 0.78 (Qwen3-VL-8B at S1-d80k) and OG up to 0.29, so identical final answers can hide opposite grounding behavior.
- Open-weight models rely on context continuity and rarely switch to tool-mediated retrieval even when provenance fails (R_tool near 0 despite WS up to 100%).
- Mandatory retrieval (Cm) improves deep pre-compaction source attribution for tool-compliant Qwen models (DeltaSC up to +78) but consistently regresses after compaction (DeltaSC -12 to -37).
Strengths
Clean factorial isolation of conversation state
Evidence objects, trunk conversation, question bank, and scoring are held fixed while only state varies (Section 3.2-3.3); each probe runs on a freshly restored checkpoint, eliminating cross-probe contamination (Section 3.4). This makes the regime claims causally interpretable.
Grounding-chain decomposition exposes outcome-only blind spots
Deterministic SC/VC/GC plus mirage index MI and OG (Section 3.5) quantify failures invisible to final-answer scoring: Qwen3-VL-8B at S1-d80k has OG=0.29 and MI=0.78, i.e., answer correctness overstates grounded correctness by 29 points (Table 2, Figure 3).
Mechanistic retrieval-path labels with compliance triage
Honest scoping and layered controls
Actionable regime separation
Weaknesses
Very narrow stimulus corpus
Single-runtime confound
Asymmetric experimental conditions
Thin inferential statistics
Mitigation costs reported but never weighed
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
- All 5 strengths & 5 weaknesses, analyzed in detail
- 5 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What is the MIRAGE benchmark?
MIRAGE is an evaluation protocol for multimodal AI agents that tests whether an agent genuinely grounds its answers in evidence it saw earlier in a conversation. It plants a fixed set of evidence objects (charts, UI screenshots) into a conversation, then asks the same questions at different conversation states - shallow context, deep pre-compaction context, and after the context has been automatically summarized/compacted - scoring answerability, source attribution, and answer value separately.
What is a 'mirage' in this paper?
A mirage is a probe where the model claims the question is answerable but fails to produce a grounded answer (wrong or missing source, or wrong value). The answer can still look correct, so outcome-only evaluation misses it; the paper's mirage index reaches 78% of answerable calls for Qwen3-VL-8B in deep pre-compaction context.
What are the main findings of MIRAGE?
Three findings across seven models: (1) deep pre-compaction context and post-compaction continuation are qualitatively different failure modes, not one smooth degradation curve; (2) open-weight models almost never switch from reading context to calling a retrieval tool even when their context-based access has clearly failed; (3) forcing tool use fixes source attribution before compaction (for models that comply) but does not help after compaction, because the stored evidence has already been degraded by the summarization step.
Is the MIRAGE code and data available?
The paper cites a public code repository (github.com/prisma-research/MIRAGE). However, full reproduction depends on proprietary API models (GPT-5, Claude-4.5-Haiku) for two backbones and for writing compaction summaries, plus provider-reported token counts and the native compaction behavior of the OpenClaw runtime, so exact replication is sensitive to API drift.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Figures are embedded as interleaved text streams (e.g., the Figure 3 MI series and Figure 4 transition counts); legend/axis mapping was inferred from captions rather than visual inspection.
- The extraction appears complete through Appendices A.1-A.6, but table formatting was flattened; a few figure counts differ slightly from the three-run table averages, likely because figures depict single runs.
- Review is based on the arXiv v1 extracted text only; the GitHub artifact was not inspected.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.