Substrate-Aware AI Agents: Execution Context as a First-Class Input
cs.AI
Quick verdict
Minimal RAM/time contracts make 3 frontier models emit leaner code (13/14 lower peak RSS, up to 3.1x faster); solid PoC but n=5, one task, no formal stats.
Summary
This paper coins "substrate blindness" for AI agents that plan without knowing the execution environment's constraints — memory limits, wall-time budgets, runtime versions — and tests whether disclosing a minimal operating contract changes what frontier models generate. Three models (Claude Opus 5, GPT-5.6-Sol, Gemini 3.7 Flash) wrote Python for an 8,000×1,024 pairwise Euclidean-distance sum under two prompts: task only, or task plus a 128 MB RAM and 10-second limit. Disclosure lowered peak process memory in 13 of 14 executable comparisons and cut mean wall time in every cohort (up to 3.1x), with audited code shifting toward blocking, float32 retention, upper-triangle traversal, and buffer reuse. Under a tighter 96 MB contract, correct-and-within-budget runs reached 4/5, 5/5, and 3/5 versus 0/5, 1/5, and 0/5 task-only. The design is admirably transparent — full prompts, raw responses, per-run profiles, and a source-linked audit are archived — but the evidence base is thin: five samples per condition, no significance testing or variance reporting, one canonical task whose optimal adaptations (blocking, float32) are textbook patterns, and heterogeneous sampling settings across the three models. It is a credible proof of concept for a real phenomenon, not yet a general result about agent planning.
Review complete
Assessment at a glance
5
risk flags identified
● high 0
● medium 3
● low 2
5 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- Failure to condition a plan on execution context ('substrate blindness') is an information-asymmetry problem: the task is visible to the agent while the operating envelope is not (Section 1).
- Disclosing a minimal 128 MB RAM + 10 s contract — with no algorithm, block size, or dtype instruction — shifts generated implementations toward bounded blocking, float32 retention, upper-triangle traversal, and buffer reuse (Sections 3.1, 4.2).
- Contract disclosure lowered measured peak memory in 13 of 14 executable index-aligned comparisons and mean wall time in all three cohorts, up to 3.1x (Section 4.1, Table 1).
- At a 96 MB contract, independently sampled disclosed cohorts achieved correct-and-within-budget outcomes of 4/5 (Claude), 5/5 (GPT), 3/5 (Gemini) versus 0/5, 1/5, 0/5 task-only (Section 4.3, Table 2).
- This constitutes a controlled proof of concept for substrate-aware agent planning, motivating a broader agenda (runtime, accelerators, tools, quotas, cost) rather than a general capability claim (Section 6).
Strengths
Clean two-condition design with honest framing
Section 3.1 explicitly states A/D suffixes are traceability identifiers, not statistically matched generations, and that cohorts are 'intentionally diverse provider configurations, not tier-matched controls' — the claims are scoped to what the design supports.
Exemplary artifact transparency
Appendix B plus reference [10] archive frozen prompts verbatim, dataset SHA-256, per-call timestamps, raw responses, per-run MaxRSS/wall-time profiles, a source-linked audit of every script, and figure-generation code — unusually complete for this genre.
Disciplined failure accounting
Effect visible at the code level, not just metrics
Consistent dose-response direction
Weaknesses
No inferential statistics anywhere
Single canonical task invites a template-recall explanation
Missing control conditions
Cross-model comparisons confounded by sampling settings
MB/MiB mismatch and MaxRSS proxy
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
The paper coins **substrate blindness**: an agent receives the task but not the execution environment's operating contract (memory, time, runtime), so it can produce plans that solve the task yet fail at deployment (Section 1, motivating examples from Kubernetes, Cloud Run, Lambda). The intervention is deliberately minimal — a single-turn prompt either bare or plus "RAM limit: 128 MB / Execution time limit: 10.0 seconds" — with no algorithm, block size, or dtype instruction, so any implementation change is attributable to the disclosed envelope. Three frontier configurations are tested: claude-opus-5, gpt-5.6-sol, gemini-3.7-flash. Claims are mostly well-scoped: the authors repeatedly disclaim statistical matching and tier-matching, and frame the work as a "controlled proof of concept" (abstract, Section 6).
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 5 strengths & 5 weaknesses, analyzed in detail
- 5 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What is 'substrate blindness' in this paper?
The authors' term for an AI agent planning a solution without knowing its execution environment's constraints — memory limits, wall-time budgets, runtime versions — so the plan may solve the task on paper yet fail when actually deployed under those limits.
Did telling the models about memory limits actually improve their code?
In this study, yes. With a 128 MB / 10-second contract disclosed, peak memory fell in 13 of 14 executable comparisons and execution ran up to ~3.1x faster. At a stricter 96 MB limit, correct-and-within-budget runs were 4/5 for Claude Opus 5, 5/5 for GPT-5.6-Sol, and 3/5 for Gemini 3.7 Flash, versus 0/5, 1/5, and 0/5 without the contract.
Is the experiment's data and code public?
The paper points to a GitHub artifact archive (manu2/Context-Aware-Agent-Experiment) containing the frozen prompts, dataset hash, raw model responses, generated scripts, per-run memory/time profiles, and figure code. Re-running the model calls requires your own API access to the three providers.
What are the main limitations of the study?
Five samples per condition with no statistical significance testing, a single pairwise-distance task whose best-known optimizations (blocking, float32) are textbook patterns, and different sampling settings for each of the three models. The authors themselves frame it as a controlled proof of concept, not a general result.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Individual per-run MaxRSS/wall-time values appear only as figure points (Figures 1, 2, A1); the text provides cohort means and counts, with raw per-run records deferred to the external GitHub archive (not inspected for this review).
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.