It is Not Yet Another Tool: Creating and Deploying an Agentic AI Companion in a Security Operations Center
Kritan Banstola, Faayed Al Faisal, Duy Dao, Ryan Irving, Daniel Lende, Xinming Ou
cs.CR · cs.AI
Quick verdict
14-month SOC field study: 90%+ draft reuse, verifiability drove adoption; time-savings rest on self-reported baselines; accuracy never independently assessed.
Summary
This ethnographic field study reports the design, deployment, and evaluation of an agentic AI companion built on locally hosted LLMs in a university Security Operations Center over 14 months. Two PhD researchers worked as tier-1 analysts (participant observation) and iteratively built a ReAct-style agent that assembles evidence from threat-intel lookups, SIEM queries, and DHCP/identity resolution and drafts triage reports; six experienced analysts then used it voluntarily during the final four months. Across 108 coded tickets, the Companion's draft text reached the closing report in 91% of cases (median reuse 96.7%), and 78 of 108 tickets reused all screenshots. Introducing verifiable screenshot evidence coincided with the largest adoption shift, though only three tickets predate it. Tickets run under analyst-written system prompts showed 98.2% median reuse versus 69.8% under the original default. Verdict and recommendation adoption were lower (67% and 69%), with analysts overriding hedges and striking about a third of recommendations to no-action. Measured median close times (7.6–20.6 minutes for the three prolific users) against self-reported manual estimates suggest 30–50% time savings. Main limitations: single site, voluntary non-random usage, no independent accuracy audit (one verified hallucination in 108 tickets), self-reported time baselines, and data that cannot be shared under NDA.
Review complete
Assessment at a glance
4
risk flags identified
● high 0
● medium 3
● low 1
5 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- Analysts reused the Companion's output in over 90% of cases: 91% of 108 closing reports reused its draft text (median draft reuse 96.7%).
- Verifiability — evidence presented as screenshots captured from authoritative sources — rather than accuracy alone, was the decisive driver of adoption.
- Analysts organically shaped the Companion through versioned system prompts, and tickets run under analyst-written prompts showed far higher draft reuse (98.2% vs 69.8% median).
- Using the Companion saved an estimated 30–50% of ticket-processing time for the three most prolific users relative to their self-reported manual baselines.
- An AI companion designed 'in the trenches' co-evolves with its users rather than being another rigid tool, and analyst shaping increases comfortable trust.
Strengths
Artifact-grounded adoption measurement, not self-report
Each of the 108 tickets links the Companion transcript to the analyst's filed closing report (Section 5.1), so adoption is measured on what analysts actually signed their name to; two-rater independent coding yields Cohen's κ = 0.965 over 2,340 binary decisions.
Three-layer adoption decomposition is a genuinely novel measurement
Section 5.4 / Figure 4 separates content (91%), verdict (67%), and recommendation (69%) adoption over the same 108 tickets, showing analysts reuse investigation prose while overriding judgment — plus Table 6's strike pattern (24/40 uncertain, 10/30 malicious recommendations struck, never a benign one).
Visible-vs-silent failure taxonomy with concrete incidents
Unusually candid limitations treatment
Actionable design finding tied to mechanism
Weaknesses
Headline productivity claim rests on uncontrolled, self-reported baselines
The central verifiability claim is supported by n=3 pre-intervention tickets
Prompt-personalization effect is confounded by self-selection
No independent accuracy audit
Zero reproducibility of the artifact
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
The paper reports a 14-month embedded field study in a university SOC where two PhD researchers worked as tier-1 analysts (participant observation) while iteratively building "the Companion": a ReAct-style LLM agent (LangChain; local GLM-4.7-Flash 30B, Qwen3.6 35B, gpt-oss 20B on a DGX Spark via Ollama) that automates evidence assembly — threat-intel lookups, SIEM hunting queries, DHCP lease and device/owner resolution — and drafts triage reports. Six manager-designated experienced analysts used it voluntarily for the final four months; 108 tickets (April 29 – July 31, 2026) link Companion transcripts to filed closing reports. Central claims: 91% content-level adoption with 96.7% median draft reuse (Section 5.2); verifiability via screenshot evidence as the decisive adoption driver (Section 5.3); higher reuse under analyst-written system prompts (98.2% vs 69.8% median, Figure 7); and 30–50% time savings for the three prolific users (Section 5.6).
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 5 strengths & 5 weaknesses, analyzed in detail
- 4 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What adoption rate did the SOC AI companion achieve?
Over 108 tickets in the final four months of deployment, 91% of analysts' closing reports reused the Companion's draft text (whole or partial), with a median draft-reuse score of 96.7%; 78 of 108 tickets reused all of its screenshots. Verdict-level agreement was lower, at 67%.
How much time did the AI companion save SOC analysts?
For the three analysts with enough tickets to characterize (n=42, 39, and 19), median assisted close times were 9.9, 7.6, and 20.6 minutes versus their own self-reported manual estimates of 10–20, ~15, and 8–30 minutes — an estimated 30–50% saving. The baselines were self-reported estimates, not measured manual times.
Why did screenshots matter so much for adopting the AI companion?
Analysts did not trust model-written claims, because text from an LLM cannot be distinguished from a hallucination without redoing the lookup. When the tool began attaching screenshots captured from the authoritative source pages, analysts could verify evidence at a glance and attach it directly to tickets; reuse jumped and utilization climbed from ~1.4% of tickets in the first four weeks to 47.9% in the final four weeks.
Can I get the code or data from this study?
No. Deployment data (transcripts, tickets, screenshots, field notes) are security-sensitive and bound by NDA and IRB protections, and the Companion's code and system prompts would disclose the SOC's detection coverage; only the codebook and code frequencies are published in the paper's appendix.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Figure-internal content (Figures 2–8) is extracted only as interleaved caption/axis-label fragments, so plotted distributions were assessed from captions and in-text statistics rather than the figures themselves; all numeric claims here come from captions, tables, and body text.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.