Can LLMs in Draft-Verify-Revise Pipelines Resolve Deictic Ambiguity?
cs.AI · cs.CL
Quick verdict
Rigorous e-value experiment shows revise-stage LLMs often misresolve deictic shifts in pipelines; convincing, but rests on just 10 synthetic stimuli.
Summary
This single-author study asks whether the revise-stage LLM in a draft-verify-revise pipeline can correctly settle perspectival disagreements caused by deictic shifts — 'previous' read either as a labeled field value or as the value in effect before a proposed change. Thirty synthetic minimal-pair stimuli (10 base examples in three conditions) hold all pipeline text constant except the fallback clause and grader feedback, isolating the ambiguity. Six closed-weight models from three providers ran at 21 reasoning-effort configurations, 20 binary-verdict trials per stimulus per experiment, in a primary experiment and an ablation removing the grader's error-classification labels; inference used anytime-valid e-values. Balanced accuracy ranged from 0.156 (GPT-5.2 without reasoning) to 1.000 for the authoring model Claude Opus 4.6†, which is flagged for stimulus contamination; the best independent configuration, Gemini 3 Pro at low reasoning effort, scored 0.965 at roughly 5% of GPT-5.2's xhigh per-trial cost. Analysis of 2,383 primary-experiment wrong verdicts indicates field-label matching rather than operational reasoning governs most errors. Main limitations, largely self-acknowledged: only 10 base examples; a meta-evaluator matching the 'Current' field could score perfectly without operational reasoning; the rationale probe classifies only wrong verdicts; and the probe audit used a same-family Anthropic model. Code, data, and per-trial outcomes are public.
Review complete
Assessment at a glance
5
risk flags identified
● high 0
● medium 2
● low 3
5 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- The revise-stage LLM of a draft-verify-revise pipeline often fails to correctly adjudicate deictic-shift disagreements, with balanced accuracy spanning 0.156 (GPT-5.2, no reasoning) to 1.000 (the authoring model) across 21 configurations on 30 synthetic stimuli.
- Reasoning effort has model-specific, non-monotonic effects: GPT-5.2 rose from 0.156 to 0.942 at xhigh effort while Gemini 3 Pro stayed above 0.94 at every level, and no adjacent-effort degradation reached significance.
- When the meta-evaluator errs, field-label matching rather than operational reasoning governs the verdict, with governing rates of 0.69% (correct-draft), 1.12% (transparent), and 2.57% (opaque) on wrong verdicts in the primary experiment.
- Removing the grader's four error-classification fields from its feedback improves accuracy for a minority of configurations (4 of 21 significant under cross-dataset tests, all favoring the ablation).
- Cost-performance spreads are extreme: Gemini 3 Pro at low reasoning effort outscored GPT-5.2 at xhigh effort for roughly 5% of the per-trial cost.
Strengths
Unusually rigorous statistical machinery
SM4 specifies per-cell Bernoulli likelihood-ratio e-values with ε-clamped seed/plug-in alternatives, testing-by-betting processes with a fixed γ=1/2 cap and sign constraints, an exactly proven closed test for each configuration's three-comparison family, e-Bonferroni families (K=15 degradation, K=6 model size, 2K two-sided cross-dataset), and beta-binomial anytime-valid confidence sequences — sequential peeking is impossible by construction.
Clean minimal-pair isolation of the phenomenon
Methods §2.3: 10 base examples × 3 conditions share the assistant's system prompt, user prompt, and proposed change verbatim; only the fallback clause and grader feedback vary, and only the meta-evaluator is an LLM, so accuracy differences are attributable to the model under test rather than upstream noise.
Transparent contamination and conflict handling
Honest confound disclosure
Full reproducibility package
Weaknesses
Tiny stimulus base limits all headline conclusions
Design shortcut undermines the 'operational reasoning' reading
Grader-stimulus internal tension contaminates the opaque condition
Same-family model dependence throughout the evaluation stack
Verdict–rationale divergence observed but never quantified
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
The paper poses a narrow, well-posed question: when a draft-verify-revise LLM pipeline passes context downstream and a deictic expression ("previous") acquires different referents at different stages, can the revise-stage LLM correctly adjudicate the disagreement? Framed as a binary verdict task (not detection, since the grader's feedback already alleges an error), the study claims (i) wide capability failure across 21 model×reasoning-effort configurations, (ii) model-specific, non-monotonic effects of reasoning effort, (iii) field-label matching as the dominant failure mode, and (iv) extreme cost-performance spread. Claims are consistently scoped to "these 30 stimuli," a discipline the Conclusion maintains.
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 5 strengths & 5 weaknesses, analyzed in detail
- 5 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What is a deictic shift in an LLM pipeline?
It is when a context-dependent expression such as 'previous', 'the response', or 'the instructions' comes to refer to different things as context passes between pipeline stages. In this paper's example, the draft-stage LLM reads 'previous dose' as the dose in effect before its proposed change (20mg), while the verify-stage LLM reads it as the value in the user prompt's field labeled 'Previous Dose' (15mg) — and the revise-stage LLM must decide which reading is correct.
Which model performed best in this study?
The stimulus-authoring model, Claude Opus 4.6, scored a perfect 1.000 balanced accuracy at medium and high reasoning effort, but it is flagged (†) because it helped write the test stimuli, so its score may reflect familiarity with its own output. The best independent configuration was Gemini 3 Pro at low reasoning effort: 0.965 in the primary experiment and 0.971 in the ablation, at about $0.0030 per trial.
Is the paper's code and dataset available?
Yes. The dataset (and its ablation variant), complete per-trial outcomes of both experiments and the rationale analysis, and the analysis code are on GitHub (oekekezie/deictic-ambiguity-companion, MIT license), with the exact version archived on Zenodo (DOI 10.5281/zenodo.22696649). Raw provider batch outputs are available from the author on request.
How much did the experiments cost to run?
The primary experiment cost $89.17 and the ablation experiment $83.23 at batch API list prices (February 2026 window, 50% batch discount included). The most expensive configuration was GPT-5.2 at xhigh reasoning effort at about $0.0556 per trial; the cheapest strong one was Gemini 3 Flash at medium effort ($0.0022 per trial, balanced accuracy 0.894).
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- The excerpt marks several '[...sampling gap...]' regions: parts of Results §3.1–3.2, Discussion §4.1–4.3, Supplementary Methods SM1–SM3, portions of SM4/SM5, the bodies of Exemplars S6–S11, and references after [33] were skipped by sampling; judgments touching those sections rest on the surrounding shown text and on Supplementary Tables S1–S2, whose data are shown.
- Figures and Supplementary Tables S3–S17 appear in the excerpt as captions/placeholders without rendered content, so table-level claims beyond S1–S2 rely on the narrative text that cites them.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.