Reflections on Trusting Trust, Revisited: Contaminating Self-Modifying AI Coding Agents with Poisoned Benchmarks
Franziska Roesner, Tadayoshi Kohno
cs.CR · cs.AI
Quick verdict
Compelling first end-to-end demos that poisoned benchmarks persistently contaminate self-modifying coding agents; small n and a modified DGM temper generality.
Summary
The paper revisits Thompson's 1984 'trusting trust' compiler backdoor in the era of self-modifying AI coding agents. The authors act as attackers who control only the benchmark feeding an agent's self-improvement loop, supplying poisoned Polyglot-style Python benchmarks whose tests force insecure code (chiefly HTTPS fetching with disabled certificate verification, because every test server presents self-signed certificates). Across three open-source systems - Darwin Godel Machine (DGM), SICA, and Hyperagents - they show evolved agents internalize the vulnerability as code-emitting tools or standing directives and then emit it on neutral held-out tasks (e.g., 30/30 vulnerable solutions for Hyperagents with Sonnet 4.5, Table 1). Contamination transfers to tasks where HTTPS is only incidental (Table 2) and persists when evolution continues on clean or even security-focused CWEval benchmarks; only poison-aware 'decontamination' benchmarks help, and fully only for Hyperagents (Table 3). Additional proofs-of-concept cover JWT signature-verification bypass (Table 4) and unsafe YAML loading (Table 7). Distilled sufficient conditions span vulnerability invisibility, forced-by-tests benchmark design, model disposition, and harness prompting (Sec 3.7). Main limitations: the DGM attack required an out-of-threat-model prompt modification; only two evolution runs per condition with ~30 held-out samples and no statistical treatment; one dominant vulnerability family; and research prototypes rather than production agents.
Review complete
Assessment at a glance
5
risk flags identified
● high 0
● medium 3
● low 2
5 strengths · 5 weaknesses · 5 author questions
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- A poisoned benchmark alone (attacker controls only the benchmark) can drive self-modifying coding agents to evolve tools or directives that write vulnerable code on neutral held-out tasks, demonstrated end-to-end on stock SICA and Hyperagents, including with frontier Sonnet 4.5.
- Contamination typically persists through continued evolution on clean benchmarks and even on a generic security-focused benchmark (CWEval plus an added CWE-295 task); only poison-aware decontamination benchmarks reduce it, fully only for Hyperagents.
- Attack success requires identifiable sufficient conditions spanning the vulnerability (functionally invisible, standard idiom), benchmark design (behaviorally forces the vulnerability with minimal cueing), model disposition (defaults to secure code), and scaffolding (does not steer away from benchmark-specific evolution).
- Stronger models are not a reliable defense: Sonnet 4.5 resisted on the (modified) DGM but was successfully contaminated on SICA and Hyperagents under benchmark reward pressure.
- LLM-based review (SICA's committee, an added security reviewer) and security-cued task naming are unreliable defenses against benchmark poisoning.
Strengths
First systematic end-to-end demonstration of the attack
Unlike prior threat-landscape papers [26, 52], the poison is empirically propagated through every stage of self-improvement (transcripts, proposal, implementation, solve-time use) with transfer to held-out tasks: 30/30 vulnerable solutions on Hyperagents+Sonnet 4.5 (Table 1) and 43/48 on the JWT variant (Table 4).
Disciplined controls and honest negative results
Paired clean/poisoned benchmarks with seed baselines (0/15, 0/48), task-shape robustness tests (Table 2), four persistence/decontamination conditions (Table 3), and transparent disclosure that stock DGM resisted until researchers modified its diagnosis prompt (Sec 3.2.1) - an out-of-threat-model change clearly flagged rather than hidden.
Cheap feasibility probes as reusable methodology
Defense analysis with genuine nuance
Responsible disclosure and release posture
Weaknesses
Underpowered, statistically unquantified experiments
Headline system (DGM) required an out-of-threat-model edit
Evaluation selects the 'best tool-bearing agent', not necessarily the shipped tip
Limited poison stealthiness and vulnerability diversity
Model-harness confounding and prototype-only scope
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
- All 5 strengths & 5 weaknesses, analyzed in detail
- 5 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What is a poisoned benchmark attack on self-modifying AI coding agents?
An attacker publishes or tampers with a benchmark used by an AI agent that improves itself against benchmarks. The tasks are designed so the agent can only pass by writing insecure code - for example, every test server presents a self-signed certificate, so an HTTPS fetcher only works if it disables certificate verification. The agent's self-improvement loop then internalizes the insecure pattern as a tool or standing instruction, so future versions of the agent write vulnerable code even on completely normal tasks.
Did the attack work on all three systems studied in the paper?
Fully within the attacker's assumed capabilities on SICA and Hyperagents (unmodified research systems), including with the frontier model Sonnet 4.5 - Hyperagents produced vulnerable solutions on 30 of 30 neutral held-out tasks. On the Darwin Godel Machine it succeeded only after the researchers modified its diagnosis prompt to encourage code-emitting tools, a change outside the attacker's threat model, so stock-DGM resilience remains an open question.
Does the contamination go away if the agent keeps evolving on clean tasks?
Mostly no. In the paper's experiments the vulnerability-producing component persisted through continued evolution on a clean counterpart benchmark and even on CWEval (a security-aware benchmark) augmented with a certificate-validation task. Only a purpose-built 'decontamination' benchmark that matched the poison's cues helped, and even then it fully removed the contamination only for Hyperagents; for DGM and SICA the effect was partial.
Is the code and data from this paper available?
Yes. The authors release well-labeled benchmark generators (deliberately not the raw poisoned benchmarks, to prevent scraping into training data), all modifications made to the three agent systems, experiment logs, and scripts at github.com/froeschele/trusting-trust-revisited. They also disclosed their findings to the authors of all three studied systems, including a confirmed answer-key leak via git history in DGM and Hyperagents.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Review is based on the extracted text; Figure 2's attack-concept diagram and the appendix vulnerability figures are analyzable only via their code listings and captions.
- The extracted text appears to end shortly after Appendix E.2 (trailing page number); any later appendix content, e.g., full decontamination-benchmark details, could not be reviewed.
- The released artifact repository (github.com/froeschele/trusting-trust-revisited) was not inspected; the reproducibility judgment rests on the paper's Open Science section.
- Closed-weight model APIs (Sonnet 4.5; Ollama Cloud-hosted gpt-oss-120b and Qwen3.5-397B) may drift or be deprecated, limiting exact replication of the reported rates.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.