Auditing Anonymous AI Models: A Four-Stage Protocol for Black-Box Identity Verification
cs.SE · cs.AI · cs.CR
Quick verdict
Forensic four-stage protocol for identifying anonymous AI models; honest scoping but end-to-end validation rests on one flagship case and tiny ablation counts.
Summary
The paper addresses black-box identity verification of anonymously launched 'stealth' LLM API endpoints. It proposes a four-stage forensic protocol: Stage 0 reconstructs launch-time declarations from Internet Archive snapshots and exposes preview-production drift; Stage 1 fingerprints configuration (context window, output ceiling, reasoning, modality) against the platform catalog; Stage 2 applies a cross-length tokenizer differential on reported prompt-token counts, requiring exactly constant differences across short and long prompts; Stage 3 adds behavioral corroboration, ranked lowest after the reveal. Validation is deliberately layered: a ten-case retrospective sample tests declaration consistency (7 exact, 2 precision differences, 1 partial, 0 counter-directional), explicitly not end-to-end identification; a single prospective flagship case (Ox Alpha) yielded a dated GLM-5.3 version-line attribution confirmed three days later by the official reveal, with the deployment variant not pre-asserted; post-reveal re-verification showed a constant 75-token differential across anonymous, public, and vendor-direct endpoints. A collision ablation shows a short-prompt-only rule admits three false positives while the multi-length rule rejects all; controlled experiments E1/E2 show Stage 2 alone is fooled by forged usage fields but a cross-stage alarm fires. Main limitations: end-to-end validation rests on one case, the retrospective sample is selection-conditioned, and no confidence intervals or statistical tests are reported.
Review complete
Assessment at a glance
5
risk flags identified
● high 0
● medium 3
● low 2
5 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- A four-stage protocol (archived declarations, catalog configuration matching, cross-length tokenizer differential, behavioral corroboration) can attribute anonymous API-served models with graded confidence and explicit 'cannot attribute' outputs.
- Cross-length constancy of prompt-token differentials is necessary evidence of tokenizer identity, while short-prompt constancy alone produces false positives (three documented collisions rejected by the multi-length rule, Table 3).
- On 10 known-identity stealth releases, launch-time declarations matched official specifications (7 exact, 2 precision-differences, 1 partial, 0 counter-directional), validating declaration-based evidence channels.
- The flagship Ox Alpha attribution to the GLM-5.3 version line was made prospectively on 2026-08-23 and confirmed by the official reveal on 2026-08-26, with a constant 75-token differential verified across anonymous, public, and vendor-direct endpoints post-reveal.
- Forged usage fields can fool Stage 2 alone (E1), so behavioral cross-checks must be retained; silent backend switches are detectable via longitudinal Stage 2 differentials (E2).
Strengths
Unusually honest claim scoping
Sections 1.3, 4.1, and 7.2 repeatedly separate declaration-consistency validation (N=10) from end-to-end identification under anonymity (N=1 flagship); the abstract itself states the retrospective sample is 'not end-to-end identification under anonymity' — a discipline most fingerprinting papers lack.
Documented failure mode with ablation
Section 3.3/Table 3 quantifies the short-prompt collision hazard with named endpoints (step-3.7-flash ∆=75, hy3 ∆=72, kimi-k3 ∆=2): short-only rule yields 2 TP/3 FP/0 TN, the mandatory multi-length rule 2 TP/0 FP/3 TN; archive-wide 2/0/6/0. Converting a community heuristic into a falsifiable criterion is the paper's core technical contribution.
Prospective chain with post-reveal triangulation
Adversarial self-testing including own weaknesses
First-class 'cannot attribute' output
Weaknesses
End-to-end validation is a single case
No statistical treatment anywhere
Stage 2 brittleness to serving nondeterminism
Converse of the criterion is not established
Evidence base is largely self-collected
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
The paper targets a real 2025-2026 phenomenon: frontier models deployed under codenames (Ox Alpha, Polaris Alpha, Quasar Alpha, etc.) on developer platforms. Its claim structure is unusually disciplined: it separates (a) declaration-consistency validation on 10 known-identity releases (Table 4: 7 exact, 2 precision-differences, 1 partial, 0 counter-directional) from (b) end-to-end identification under anonymity, demonstrated on exactly one prospective case (Ox Alpha, Section 5.1). Sections 4.1 and 7.2 restate this scoping; the abstract states it. This honesty is the paper's best feature — reviewers should hold it to the narrower claim. Supporting numbers: 422→6→1 catalog shrinkage (Fig. 3); constant ∆=75 tokenizer differential across five canonical probes and, post-reveal, across three serving endpoints (Section 5.1).
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 5 strengths & 5 weaknesses, analyzed in detail
- 5 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
How can you identify an anonymous AI model using only API access?
The paper infers identity from deployment properties a stealth operator cannot easily fake: archived launch-time configuration from the Internet Archive, platform catalog fields (context window, output limits, modality), and the difference in reported prompt-token counts between endpoints on identical prompts. If that difference stays exactly constant across short and long prompts, the two endpoints very likely share a tokenizer, which implies the same model family; behavioral probes only corroborate.
Did the method correctly identify the Ox Alpha stealth model?
Yes at family and version-line level: the author's 2026-08-23 analysis attributed Ox Alpha to Zhipu's GLM-5.3 line, and the official reveal on 2026-08-26 confirmed it as GLM-5.3-Flash. The specific Flash variant was not predicted in advance. After the reveal, a constant 75-token prompt-count differential was verified across the anonymous gateway, the public endpoint, and the vendor's official API.
What are the main limitations of this identity-audit protocol?
End-to-end identification under anonymity was validated on a single flagship case; the 10-case retrospective sample only validates that launch declarations match official specs, not identification accuracy. No statistical error estimates are given, the tokenizer stage can be fooled by a proxy that forges usage fields (a cross-stage alarm is the countermeasure), and platforms that do not expose token counts (like OpenCode Zen in the paper) limit applicability.
Is there code or data available for this paper?
Yes: the author states a standard-library-only Python implementation (audit-tool implementing Stages 1-3), an auditor worksheet template, Stage 0 extraction sheets, and dated measurement logs (catalog snapshots, tokenizer-differential JSONL, E1/E2 experiment logs) included as supplementary material with a public Zenodo deposit (DOI 10.5281/zenodo.22210928). Stage 0 is a documented procedure, not a packaged crawler.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Figures 1-4 are referenced only by caption; their visual content is not recoverable from the text extraction, so figure-based judgments rely on in-text descriptions and tables.
- Supplementary materials S1-S4 (case corpus, worksheet, Stage 0 extracts, experiment logs) are referenced but not part of the provided text; claims resting solely on them could not be checked.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.