Who Is Behind the Harness? Fingerprinting LLMs through Agentic Behavior
Chuyi Wang, Xiaohui Xie, Tongze Wang, Fangchen Luo, Yong Cui
cs.CR · cs.AI · cs.SE
Quick verdict
主动式黑盒大模型指纹识别，通过编码智能体探针实现；在 36 个模型上表现出色，并进行了消融实验，但仅限于闭集（closed-set）场景，且规避测试（evasion untested）尚未验证。
Summary
本文提出了 LIDAR（LLM Identification from Decisions and Actions at Runtime），这是一种主动式黑盒指纹识别方法，用于识别哪个 LLM 在为编码智能体端点提供服务，其动机是解决广告宣传的身份背后存在的静默模型替换问题。三组成对的编码探针——VERIFY、RECOVER 和 RESOLVE——在保持修复任务不变的前提下，改变一个受控条件（如验证要求、瞬时测试失败或规范与测试冲突），从而暴露出诸如编辑后检查、故障恢复和冲突解决等决策。每个样本的六个轨迹被转换为 756 个实例级和 202 个分布级特征，并使用 Jeffreys 平滑概率标识符与冻结的特定于测试框架（harness-specific）的参考进行评分，仅使用对工具事件、仓库状态和测试结果的黑盒访问。在来自 7 个系列的 36 个模型和两个测试框架上，论文报告的 Top-1 准确率在 mini-swe-agent 上为 95.13%（MRR 0.9736），在 OpenCode 上为 88.36%（MRR 0.9336），在匹配采样预算的情况下优于 LLMmap、MET、ZeroPrint 和 One Token；消融实验证实，每个探针对、两个 A/B 变体以及两个特征级别均有贡献，并且在六个模型的子集上，两种提供方端的指令操作最多使 LIDAR 的 Top-1 降低了 11.11 个百分点。主要局限：评估仅限于闭集（虽然陈述了开集扩展，但未经测试），鲁棒性测试未涵盖主动规避或模仿攻击，在朴素独立性假设下，958 个特征远超每类 6 个的注册样本数量，且未声明代码或数据的公开。
Review complete
Assessment at a glance
6
risk flags identified
● high 1
● medium 3
● low 2
4 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- Agent execution behavior — tool use, action order, verification, recovery, and repository outcomes — carries model-identity evidence beyond final text outputs (Sections 1, 6.5).
- LIDAR identifies 36 models from seven families with Top-1 accuracy of 95.13% (MRR 0.9736) on mini-swe-agent and 88.36% (MRR 0.9336) on OpenCode, achieving the best mean in all six metric–harness comparisons against four baselines (Table 4).
- All three probe pairs, both A/B variants, and both instance- and distribution-level feature views contribute to identification (Table 5, Figure 3).
- LIDAR degrades by at most 11.11pp Top-1 under two provider-side system-instruction manipulations applied to queries on a six-model subset, while every baseline loses accuracy in at least one condition (Table 6).
Strengths
Carefully scoped threat model with an original observation unit
Section 2 formalizes model substitution (Eq. 2) and explicitly disclaims remote attestation, open-set recognition, and mixed-routing claims before any evaluation; Section 1's insight that the harness is not only an obstacle but a new source of identity evidence reframes fingerprinting around agent decisions rather than final text.
Controlled paired-probe design with locally verifiable outcomes
Table 2's VERIFY/RECOVER/RESOLVE pairs hold the repair goal fixed while varying one condition; outcomes (exit codes, diffs, verifier-controlled checks) are recorded on the user's host, so evidence does not depend on model self-reports. Retaining the ordered pair (A,B) rather than B−A distinguishes models that always verify from those that verify only when instructed.
Budget-matched, comprehensive evaluation
Transparent, self-limiting reporting
Weaknesses
Closed-set identification undercuts the auditing headline
Robustness excludes the adversarial behaviors the threat model permits
Extreme p>>n regime with unquantified uncertainty
Serving-stack confound not disentangled
Reproducibility and verifiability gaps
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
The paper proposes LIDAR (LLM Identification from Decisions and Actions at Runtime), an active black-box method for identifying which LLM powers a coding-agent CLI endpoint, motivated by model-substitution auditing: a provider, relay, or router may silently serve a different model than advertised (Section 2, Figure 1). Three paired coding probes (VERIFY, RECOVER, RESOLVE; Table 2) hold a repair task fixed while changing one controlled condition — a verification requirement, an injected transient test failure, a specification–test conflict — so identity evidence comes from how the agent decides (post-edit checking, failure recovery, conflict resolution) rather than from final text. Headline claims: Top-1 accuracy of 95.13% (MRR 0.9736) under mini-swe-agent and 88.36% (MRR 0.9336) under OpenCode over 36 models from seven families (Table 4); superiority over LLMmap, MET, ZeroPrint, and One Token in all six metric–harness comparisons; and verified contributions of every probe pair, both A/B variants, and both feature levels (Table 5). The threat model explicitly scopes the verdict as closed-set behavioral evidence, not remote attestation.
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 4 strengths & 5 weaknesses, analyzed in detail
- 6 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What is LIDAR and how does it fingerprint an LLM?
LIDAR is a black-box method that identifies which large language model is serving a coding-agent endpoint. It runs three pairs of small coding tasks (checking a change after editing, recovering from a deliberately injected transient test failure, and resolving a conflict between the specification and a stale test), records how the agent behaves — which tools it uses, in what order, whether it verifies its edits, how it recovers — and compares those trajectory features against pre-collected references for each known model, using a simple probability model with Jeffreys smoothing. It needs no access to model weights or logits, only the agent's observable actions and local test outcomes.
How accurate is LIDAR at identifying models?
According to the paper, across 36 models from seven families and two coding-agent harnesses, LIDAR achieves Top-1 accuracy of 95.13% (MRR 0.9736) on mini-swe-agent and 88.36% (MRR 0.9336) on OpenCode, beating four prior fingerprinting methods (LLMmap, MET, ZeroPrint, One Token) under equal sampling budgets. Note that the evaluation is closed-set: it always picks among 36 registered models and cannot declare an unknown model, since the open-set extension described in the paper was not experimentally evaluated.
Can LIDAR detect if my API provider secretly swapped the model?
Partially, with caveats. If the substitute model is among the enrolled references, LIDAR can flag a mismatch between the best-matching model and the claimed identity. If the substitute is a model that was never enrolled, the evaluated version still forces a verdict among registered labels; the authors describe (but do not test) a threshold-based extension for rejecting unknowns. The robustness study also only covers benign provider-side instruction changes — a provider actively trying to imitate another model's behavior was not tested.
Does LIDAR need model weights, logits, or provider cooperation?
No. It only requires the ability to run a coding-agent CLI on your own machine against the remote endpoint and to observe local, verifiable events: tool calls, test exit codes, repository diffs, and final responses. Enrollment references are collected once per claimed model and harness and then frozen; query samples never update them.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Figures 1-5 were analyzed only from their embedded textual descriptions; pixel content, exact layouts, and Figure 5's full mechanism tables could not be inspected directly.
- The 36-model panel (Table 7) includes model identities that cannot be externally verified from the provided text; identification results inherit whatever serving conditions the authors used.
- Any appendices beyond Table 7 and portions of Section 5.2's formulas were reconstructed from possibly lossy text extraction; minor notation details may be imprecise.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.