After the Party: Growth, Governance, and Security Scanning in the OpenClaw Agent Skill Ecosystem
cs.SE · cs.AI · cs.CY
Quick verdict
Careful audit of the OpenClaw/ClawHub boom; unstable-metadata and scanner-disagreement findings are credible, but the human reference audit is small.
Summary
This empirical study measures what the 2026 OpenClaw/ClawHub agent-skill boom left behind, using the core repository's Git history (68,858 commits), 94,248 GitHub issues/pull requests, and three registry snapshots (March, June, July 2026). RQ1 sizes the boom: registry stock nearly doubled (33,399 to 65,175 listings, +95.14% in 91 days), 63.25% of June listings were created in March-April, and downloads are concentrated (top 10% hold 46.93%, Gini 0.528). RQ2 shows seven pre-declared metadata/artifact features fail as stable predictors of continued visibility: only three survive multiplicity correction, the download effect reverses sign (-0.194 to +0.204), age adjustment leaves only has-scripts positive, and none survive pre-cutoff cohort restriction. RQ3 documents a reviewability gap: 77.86% of listings have zero stars and comments while 85.06% of 64,324 evaluable artifacts match at least one of twelve hand-authored privilege rules. RQ4 audits the deployed LLM, static-analysis, and VirusTotal scanners against a blinded, two-annotator adjudicated reference on a SHA-256-verified 276-case pool (180 sampled): they disagree on 23,702 of 61,990 jointly covered listings, and weighted sensitivity spans 21.67-61.06%. Main limitations: a small, selection-filtered audit pool; an unvalidated privilege detector; no inter-annotator reliability statistics; a weak, short-horizon visibility outcome; and two data sources withdrawn mid-study.
Review complete
Assessment at a glance
5
risk flags identified
● high 1
● medium 3
● low 1
5 strengths · 5 weaknesses · 5 author questions
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- The observable ClawHub registry stock nearly doubled in 91 days (33,399 to 65,175 listings), with downloads heavily concentrated: the top 10% of listings hold 46.93% of downloads (Gini 0.528).
- No simple metadata or artifact feature is a stable predictor of continued listing visibility: 0 of 7 associations survive pre-cutoff cohort restriction, and the download association reverses sign (-0.194 to +0.204) across windows.
- A reviewability gap exists: 77.86% of listings have zero stars and zero comments while 85.06% of 64,324 evaluable artifacts carry evidence for at least one of twelve privileged capabilities.
- The three deployed scanners disagree on 23,702 of 61,990 jointly covered listings and reach weighted sensitivity of only 21.67%-61.06% against an adjudicated human reference, so neither any single scanner nor majority vote can stand in for ground truth.
Strengths
Measurement discipline unusual for registry studies
Units are reconciled to stable list-page IDs with fail-closed checks, tri-state present/absent/unknown semantics are preserved throughout (Sec. II-III), the 95.14% stock growth is confirmed across four identity definitions (94.83-95.14%), a separate verifier program recomputed every release, and the replication package documents inputs with SHA-256 hashes.
RQ2 inference is pre-declared and honestly null
Seven features declared up front with four exclusion families justified; Mann-Whitney U with rank-biserial effects under within-cohort BH correction, per-SD age-adjusted logistic models, and five pre-declared stability diagnostics. The paper reports its own download sign reversal (-0.194 to +0.204, Table I) and the 0/7 pre-cutoff failure rather than burying them.
First adjudicated scanner audit for this registry
Governance-relevant construct separation
Transparency about a vanishing observation surface
Weaknesses
Reference standard is two-person consensus without reliability statistics
Audit pool is small and doubly filtered
RQ2 outcome is weak and short-horizon
Privilege detector unvalidated despite carrying the RQ3 headline
March cohort is reconstructed, not directly observed
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
- All 5 strengths & 5 weaknesses, analyzed in detail
- 5 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What is the paper 'After the Party: Growth, Governance, and Security Scanning in the OpenClaw Agent Skill Ecosystem' about?
It is an empirical study of the OpenClaw AI agent and its public skill registry ClawHub during the 2026 boom. Using three registry snapshots plus Git and GitHub data, it measures how fast the ecosystem grew, whether registry metadata (downloads, stars, versions, file counts) reliably predicts whether skills stay listed, how much human review the artifacts actually receive, and how well the registry's three deployed security scanners perform against human judgment.
Do the security scanners on ClawHub work well according to this study?
The study finds the three scanners cover 97-99% of listings individually, but they disagree on 23,702 of the 61,990 listings all three cover. Against a blinded, human-adjudicated reference of 276 file-verified cases, weighted sensitivity ranged from 21.67% (static analysis) to 61.06% (the LLM scanner), and among listings no scanner flagged, 24.16% of the weighted mass was still judged concerning by humans. The conclusion is that no single scanner or majority vote can substitute for ground truth.
Does the finding that 85.06% of OpenClaw skills carry 'privilege evidence' mean most skills are dangerous?
No. The 85.06% figure means that share of readable skill files contains text matching rules for privileged capabilities, such as shell execution (58.08%) or network access (57.06%). This is evidence about what the artifact text mentions or declares, not proof of malicious behavior, exploitation, or runtime effects, and the authors explicitly interpret rule matches under a frozen rule set rather than as intent or behavior. The detector itself has not yet been validated against human labels, which the paper lists as future work.
Is the paper's data and code available for replication?
Yes. The authors publish an anonymized replication package on Zenodo (DOI 10.5281/zenodo.21469516) documenting every data field, loading rule, and input file with SHA-256 hashes, and a separate verifier program recomputed each release. However, two primary sources - the registry's public Git repository and comment bodies - were withdrawn during the study, so live replication of those parts is impossible; the archived snapshots are the evidence of record.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Figures 2-5 were parsed only from embedded caption text; visual details of curves and overlaps could not be independently verified.
- The extraction appears complete but contains minor OCR artifacts (e.g., 'vot e', stray quotes), so full-page truncation cannot be entirely ruled out.
- The Zenodo replication package was not inspected; reproducibility claims rest on the paper's own description.
- Scanner identities (e.g., SkillSpector) and platform internals are described only as the paper reports them; no external verification was performed.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.