LEGIT: Credentialing Protocol for Trustworthy AI Agent Marketplaces
cs.AI
Quick verdict
LEGIT proposes signed credentials binding agent quality and cost to config and budget; careful experiments, but allocation and feedback checks untested.
Summary
The paper introduces LEGIT, a three-layer protocol for AI agent marketplaces combining certification, reputation, and auction-based allocation. Layer 1 issues signed credentials binding measured task quality and cost per solved task to the agent configuration (model dependencies, harness, tools, parameters), task domain, token budget, evidence, and expiry. Layer 2 maintains Beta reputation with geometric discounting plus identity deposits; Layer 3 proposes a scoring auction combining quality commitment, certified cost, reputation, and price with a second-score payment rule. Empirically, the authors evaluate 18 configurations (three models x six harnesses) on 127 GAIA tasks at 8,000-token budgets, plus retrieval runs at 16,000 tokens and 127 MATH-500 tasks. Model effects are robust (adjusted permutation p=0.00020; gpt-5.6-sol solves 55/127 vs 43 for gpt-5.6-luna at baseline), while harness effects are insignificant after Bonferroni correction; cost per solved task varies up to 4.95x within a model, and retrieval rankings flip with the token budget. Repeatability runs show up to ~20% SD in solved counts even at temperature 0. A Sybil analysis derives closed-form interaction counts and deposit/fee requirements under stated assumptions. Main limitations: the allocation and payment rules are never evaluated end-to-end, reputation feedback authenticity is assumed rather than verified, the harness grid mixes invocation-cache conditions, and no code release is described.
Review complete
Assessment at a glance
5
risk flags identified
● high 0
● medium 2
● low 3
5 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- Agent quality and cost measurements are only comparable when bound to the tested configuration, task domain, and token budget, as shown by budget-dependent ranking reversals and 1.43x-4.95x within-model cost spreads at similar success rates.
- Model choice has a robust effect on GAIA task success (adjusted permutation p=0.00020; sol 55/127 vs luna 43/127 at baseline), whereas harness variations show no significant quality effect after Bonferroni correction in the tested grid.
- A signed structured credential (Layer 1) can connect certification to discounted-Beta reputation (Layer 2) and scoring-auction allocation (Layer 3) in a coherent protocol for agent marketplaces.
- Under the stated Sybil model, deposits plus non-recoverable fees impose quantifiable funding barriers (e.g., 23 perfect-feedback interactions to reach reputation 0.99 at eta=0.9; 34.1% of 0.8-reliability attackers fail within 5,000 interactions).
Strengths
Disciplined, honest statistical reporting
A pre-stated decision criterion (kappa-hat >= 1.2 with 95% CI excluding 1.0) plus retrospective exact paired binomial tests and permutation omnibus tests with explicit Bonferroni families (Appendix D, Tables 10-11); the paper plainly reports that all 15 within-model harness contrasts are non-significant instead of leaning on unadjusted intervals.
Experiments directly motivate each credential field
Budget-dependent ranking flips (Fig. 3a: luna web search is 0.45x baseline quality at 8,000 tokens but 1.26x at 16,000), model-dependent tool benefits (search: luna 38->48 solved, sol 54->52), and within-model cost spreads of 1.43x-4.95x (Fig. 4) empirically justify recording configuration, domain, budget, quality, and cost together.
Repeatability quantified rather than ignored
Verifiable Sybil-cost analysis
Explicit proposed-vs-evaluated scoping
Weaknesses
The marketplace mechanism itself is never exercised
Reputation integrity is deferred, not solved
Harness grid has a runtime-condition confound and thin power
Narrow domain coverage versus protocol generality
No code/data release; judge calibration asserted, not tabulated
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
LEGIT (arXiv 2609.21325) is a three-layer protocol for AI agent marketplaces. Layer 1 issues signed credential records (Ed25519) binding mean task quality and cost per solved task (CoP, Eqs. 1-2) to an agent configuration (model dependencies, harness, tool suite, parameters), a task domain, a token budget, an evidence record, and an expiry. Layer 2 maintains a geometrically discounted Beta reputation (Eqs. 3-4) per identity, backed by deposits. Layer 3 proposes a Che-style scoring auction (Eqs. 5-6) over quality commitment, certified cost, reputation, and price with a second-score payment rule. The central claim — that quality and cost measurements are only comparable when bound to configuration, domain, and budget — is supported by the paper's own controlled experiments rather than asserted a priori. Importantly, the allocation layer is labeled 'proposed' throughout; end-to-end marketplace behavior is explicitly not validated (Sections 3.3, 5).
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 5 strengths & 5 weaknesses, analyzed in detail
- 5 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What is the LEGIT protocol?
LEGIT is a proposed three-layer protocol for AI agent marketplaces. Layer 1 issues signed credentials that bind measured task quality and cost per solved task to a specific agent configuration (model, harness, tools, parameters), task domain, token budget, evidence, and expiry. Layer 2 tracks reputation with a geometrically discounted Beta model plus identity deposits. Layer 3 allocates tasks through a scoring auction combining quality commitment, certified cost, reputation, and bid price.
Do fancier agent harnesses (ReAct, reflection, memory) improve performance?
Not in this paper's controlled experiments. Across 18 configurations on 127 GAIA tasks at an 8,000-token budget, no harness variant significantly beat its model's baseline after multiple-comparison correction (all 15 paired tests non-significant), although harnesses changed cost per solved task by 1.43x to 4.95x within a model. Model choice mattered far more: gpt-5.6-sol solved 55/127 versus 43 for gpt-5.6-luna with the same baseline harness.
How expensive is it to fake reputation with fake identities (a Sybil attack)?
Under the paper's model, each fake identity must lock a deposit and pay non-recoverable fees per interaction. With discount factor eta=0.9 and perfect positive feedback, reaching a reputation of 0.99 takes 23 interactions; with only 80%-reliable positive feedback, 34.1% of simulated attackers never reach that target within 5,000 interactions. The authors caution this quantifies funding requirements but does not prove attacks unprofitable or detect Sybil identities.
Is code or data available for this paper?
No public repository is referenced. The paper describes an evaluation system with Ed25519-signed manifests, fixed random seeds, pinned model snapshots, and price snapshots, which supports partial reimplementation, but neither code nor the task-level outcome matrix is stated to be released.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Figures 1-7 appear in the extraction only as axis labels and captions; plot-level details were inferred from captions and surrounding text, not from rendered images.
- Model names (gpt-5.6-luna, gpt-5.6-sol, claude-haiku-4-5) are taken as stated; no independent verification of provider identities is possible from the text.
- Some appendix table cells (e.g., Table 8 fee columns) were cross-checked against narrative text; minor extraction garbling of figure text is possible.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.