Plan Pointers and Record-Directive Form in Budgeted Verification of Inherited Agent Memory
cs.IR · cs.AI · cs.CL
Quick verdict
16 registered studies (179,352 attempts) show directive form steers agent memory verification model-specifically; exemplary rigor, thin generality.
Summary
Across sixteen preregistered studies (179,352 attempts, up to fifteen models), the paper measures how the form of a directive written into an inherited agent memory store — bare record id, length-matched natural-language criterion, or both combined — changes which archived source record an agent pulls under a one-credit verification budget. Key results: on six direct-provider models a criterion beat a bare id by +35.0 points [31.2, 38.8] (D) but failed its superiority rule on a disjoint OpenRouter panel (E, +7.2 [0.0, 14.4]); appending the id cancelled the criterion on Claude endpoints (Opus 5: 40/40 to 0/40; F/F-x), sometimes restored by a ratification line (+96.0, J); byte-controlled edits (G/G′) attribute cancellation to exact strings; a one-character plan pointer moved +78.0 points, prospectively replicated at +81.7 (B/B′). On 72 model-generated worlds the two registered signatures held (−41.3 Opus 5; −82.3 Fable 5.1; K2) while Haiku 4.5 reversed (+75.0). With Meta-SecAlign at rising gains, the 8B composite-minus-criterion attenuation met its rule at registered power on 337 families (−16.6 [−19.4, −13.8]; K5) but a SecAlign++ adapter did not (−11.8). All results are descriptive: no mechanism claims, fixed panels, one scenario lineage, no familywise correction. Integrity reporting (a corrected headline, a quarantined contaminated run) is unusually candid.
Review complete
Assessment at a glance
5
risk flags identified
● high 0
● medium 3
● low 2
5 strengths · 4 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- The form of a verification directive with fixed referent — bare id, length-matched criterion, or composite — changed where the request went by more than the 10-point margin in 7 of Study F's twelve registered contrast-group entries, in model-specific directions.
- On three named Claude endpoints the composite of criterion and id produced lower target rates than the criterion alone (Opus 5: 40/40 to 0/40, Study F-x), and Study G attributes the cancellation to exact appended strings rather than any string class.
- A one-character ACTIVE PLAN ID pointer moved the request +78.0 points (0.929 of a plan sentence's effect), prospectively replicated at +81.7 under a registered rule (Study B′).
- On 72 generated worlds in 36 families the two registered signatures held (composite −41.3 below criterion on Opus 5, −82.3 on Fable 5.1) while Haiku 4.5 reversed (+75.0) and five other endpoints lay near zero (Study K2).
- At registered power on 337 families the 8B Meta-SecAlign ladder satisfied the registered attenuation criterion (change −16.6 [−19.4, −13.8]) while a SecAlign++ adapter did not (−11.8 [−14.3, −9.3]); all statuses are explicitly not statements that contrasts were unchanged (Study K5).
Strengths
End-to-end provenance discipline
SHA-256 manifests, OSF deposits, OpenTimestamps attestations, frozen analyzers and an independent re-implementation reporting zero differences (K3-K5); a §12 deviations catalogue that discloses a corrected headline (Study B) and a quarantined contaminated run (Study F) rather than hiding them.
Registered claim discipline
Verdict rules fixed before each run with a ±10-point margin; 'not met' is repeatedly glossed as not implying 'unchanged' (§11.3, §11.4); §1 and §14 explicitly enumerate what is not claimed — no mechanism, no length separability, no decision improvement.
Heterogeneity reported, not averaged away
Contamination control as a reusable contribution
Prospective replications of the fragile results
Weaknesses
Descriptive to a fault; no mechanism or design guidance
External validity confined to named endpoints
No familywise error control across a large interval census
Verdict-status bookkeeping invites misreading
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
This is a measurement paper, not a systems or methods paper: sixteen preregistered studies (179,352 attempts; up to fifteen models per study) on one instrument lineage — an inherited store of one-line memories with provenance links, a k=1 verification budget, and the deterministic endpoint V73 ('was the target record named first'). The manipulated variable is the byte-exact form of a VERIFY PRIORITY directive: none, bare id ('memory 73'), a criterion length-matched to the character ('a record that could invalidate a direction you are not currently planning to take'), or their composite in two layouts (IL/IF). The authors claim only descriptive contrasts with registered intervals and verdict rules — explicitly not decision improvement, not mechanism, not length separability (registered as bundled, §3). The procedural contribution (a per-cell prompt-token invariance gate born from a real contamination, §12/Appendix B) is genuinely useful for anyone preregistering hosted-model studies.
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 5 strengths & 4 weaknesses, analyzed in detail
- 5 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What is this paper about?
It reports sixteen preregistered experiments testing whether the wording of a directive placed in an AI agent's inherited memory store — a bare record ID, a natural-language criterion naming the same record, or both together — changes which archived record the agent chooses to verify when its budget allows checking only one record before acting.
What is the main finding?
Directive form matters, but model-specifically: on six direct-provider models a criterion beat a bare id by +35.0 points, while on three Claude models appending the id to the criterion cancelled it (Opus 5: 40/40 down to 0/40). On 72 model-generated worlds the cancellation signatures held on Opus 5 (−41.3) and Fable 5.1 (−82.3) but reversed on Haiku 4.5 (+75.0). No mechanism is claimed.
Is the research preregistered and reproducible?
Yes — every study was frozen before running (SHA-256 manifests, OSF deposits, OpenTimestamps proofs) with verdict rules fixed in advance, and a Zenodo release with per-episode data, analyzers and an independent re-implementation is described. Deviations, including a corrected headline result and a quarantined contaminated run, are disclosed in the paper itself.
Does the paper explain why appending the ID cancels the criterion?
No. All results are explicitly descriptive. The instrument cannot separate copying versus dereferencing, length versus content, or a trained disposition, and the authors claim no mechanism — byte-controlled edits show the effect belongs to exact strings, not to any general class of suffixes.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- The text is a sampled excerpt with '[...sampling gap...]' markers: full procedures of Studies J and H2 (§7, §8.1), parts of §11.2-11.3 and most appendix tables fall in gaps and were assessed via the abstract, §1 findings and §14, which reprint the key numbers.
- Tables 4-44 are only partially visible; per-cell intervals quoted in this review are those printed in the visible narrative text.
- Study K3/K4 section bodies lie partly in the sampling gap; their ladder values were taken from the abstract, §11.4 and §13 where they are reprinted.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.