Can AI Agents Detect and Repair Artifact Drift in Network Experiments?
Tianzhu Zhang, Weichen Tao, Changgang Zheng, Yusheng Zheng, Long Chen, Xiaoyi Fan, Meikang Qiu
cs.NI · cs.AI · eess.SY
Quick verdict
NetArtifactBench (52 instances) tests AI-agent repair of network experiment records; rigorous scoring finds 65.3% pass but under 30% on cross-artifact repairs.
Summary
The paper introduces artifact integrity — the requirement that claims in a network experiment record remain supported by evidence, scoped correctly, and traceable — and presents NetArtifactBench, a benchmark of 52 instances built from public Batfish, Zeek, P4C, and Open vSwitch artifacts with controlled, injected inconsistencies spanning five structural tiers, from single exposed contradictions to unstated cross-artifact relations. Twenty-three agent configurations — combinations of Codex CLI, Cursor Agent, and OpenCode runtimes with backends including GPT-5.x, Claude, Gemini, Grok, and open-weight models — are each run five times on every instance, yielding 5,980 outputs scored by a deterministic contract that requires complete diagnosis, cross-artifact propagation, preservation of supported claims, and parseable, traceable outputs. The overall contract pass rate is 65.3%, but performance collapses with structural complexity: from 98.9% (Tier 1) to 26.3% (Tier 5) for Codex, 93.8% to 28.4% for Cursor, and 60.8% to 17.0% for OpenCode; the strongest configurations reach only about 81% (GPT-5.5/xhigh under Codex, Claude Opus 4.8 under Cursor). Failure analysis of 2,076 failed outputs shows 68.5% leave diagnoses or repairs incomplete while only 3.9% assert the inverse of the seeded claim, and 27.7% of configuration–instance pairs flip between pass and fail across repetitions. Limitations include synthetic, author-designed violations, author-constructed oracles without independent review, an unstated per-tier instance distribution, and no stated release of benchmark code or instances.
Review complete
Assessment at a glance
5
risk flags identified
● high 0
● medium 3
● low 2
5 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- Artifact integrity — claims supported by evidence, confined to its scope, and traceable — is a necessary property of agent-modified experiment records that existing outcome-centered benchmarks do not measure.
- NetArtifactBench operationalizes this via 52 source-grounded instances with injected inconsistencies across five structural tiers and a deterministic contract scorer.
- Across 23 agent configurations and 5,980 outputs, the average contract pass rate is 65.3%, but no runtime exceeds 30% on Tier-5 repairs requiring inferred cross-artifact relations (Codex 26.3%, Cursor 28.4%, OpenCode 17.0%).
- Agents typically satisfy individual obligations without jointly completing the repair (Tier-5 label accuracy up to 96.2% for Codex vs 26.3% pass), and 27.7% of configuration–instance pairs flip between pass and fail across repetitions.
Strengths
Novel evaluation target with a formal model
Section 3 formalizes record-level integrity as a claim–evidence–scope graph with valid support paths (scope_eff(c) ⊆ ctx(p)), moving beyond outcome-centric benchmarks (NetConfEval, NetAgentBench, Cornetto) toward auditing what an agent's edits leave behind.
Careful, validated instance construction
Fig. 2 shows a freeze-after-validation gate: reference repairs must pass and seeded incorrect repairs must fail, hidden annotations add no evidence and meet the agent only at the deterministic scorer — a clean agent/oracle separation across 52 instances from four real projects (Batfish, Zeek, P4C, Open vSwitch).
Internally consistent, large-scale evaluation
Honest partial-credit analysis localizes the failure
Instability reported rather than hidden
Weaknesses
Rubric circularity: authors design drift, oracle, and scoring
All-or-nothing headline metric
Unknown per-tier instance distribution
Cross-runtime aggregates conflate model portfolios
No availability statement
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
The paper targets a gap in how AI agents on network systems are judged: not whether the immediate task succeeds, but whether the experiment record the agent edits remains trustworthy. It defines **artifact integrity** (claims supported by evidence, confined to evidence-established scope, traceable through artifacts), formalizes it as a claim–evidence–scope graph with valid support paths (Section 3), and instantiates it as **NetArtifactBench**: 52 instances from public Batfish, Zeek, P4C, and Open vSwitch artifacts, with injected inconsistencies across five structural tiers (Table 1, from single exposed contradictions to inferred cross-artifact relations). Central claims: existing benchmarks are outcome-centered; 23 agent configurations over Codex CLI, Cursor Agent, and OpenCode pass 65.3% of 5,980 contract evaluations overall but none exceeds 30% on Tier-5 repairs; agents satisfy individual obligations while failing joint completion, with outputs unstable across repetitions. A five-mode failure taxonomy (Fig. 1: residual inconsistency, unsupported replacement, support damage, lost traceability, invalid package) grounds the design.
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 5 strengths & 5 weaknesses, analyzed in detail
- 5 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What is artifact integrity in this paper?
It is the property that every claim in a network experiment record remains supported by available evidence, stays within the scope that evidence establishes, and remains traceable to the artifacts that justify it — so downstream tools or agents can audit what changed and why.
What are NetArtifactBench's main findings?
Across 23 agent configurations, 52 instances, and 5,980 outputs, 65.3% passed the full repair contract, but pass rates fall to 17–28% when repairs require inferring unstated cross-artifact relations (Tier 5). Agents usually detect problems (label accuracy up to 96.2% on Tier 5 for Codex) but rarely complete every repair obligation jointly, and about 27.7% of configuration–instance pairs give mixed results across repeated runs.
Which AI agents performed best on NetArtifactBench?
GPT-5.5 at xhigh reasoning effort under Codex CLI reached 81.9% and Claude Opus 4.8 under Cursor Agent reached 80.8% contract pass. OpenCode's best was DeepSeek V4 Flash at 72.3%, while local Qwen3 8B managed only 1.2%.
Is the NetArtifactBench benchmark publicly available?
The paper does not state a public release of the instances, contracts, or scorer. The instances are derived from public artifacts of the Batfish, Zeek, P4C, and Open vSwitch projects, but the injected mutations, oracle annotations, and evaluation harness availability are not announced in the text.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Figure 1 and Figure 2 panels survive only as fragmented caption/graphic text in the extraction; the review relies on the body prose (Sections 2 and 4) that describes them, which was sufficient for all judgments made.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.