terms.txt: A Consent and Compensation Protocol for Agentic Web Access
cs.NI · cs.AI · cs.CR · cs.CY
Quick verdict
Specifies terms.txt for per-purpose machine-access terms with a signed origin exchange; well-reasoned and honest, but validated by loopback microbenchmarks.
Summary
The paper argues that the web's implicit crawl-for-referral bargain has broken under AI crawlers: assembled operator data show automated traffic at 51–57.5% of requests, training dominating classified crawler fetches, and Anthropic's crawlers once requesting roughly 70,900 pages per referred visit versus about 5 for search. It shows robots.txt cannot express identity, purpose, terms, or price and is unverifiable in practice. The contribution is terms.txt, a robots.txt-style well-known file granting per-path, per-purpose allow/charge/deny rules, composed with an origin-enforced HTTP exchange: Web Bot Auth signatures (RFC 9421), signed Access-Intent headers, agent-bound pseudonymous delegation tokens, payment vouchers, HTTP 402 negotiation, and signed hash-chained receipts. The design explicitly separates what is enforced before delivery, auditable after delivery, and contractual after content use — it does not claim to prevent training on delivered bytes. A ~600-line dependency-free Node.js implementation adds 0.20 ms (identity only) to 0.65 ms (charged path with voucher) per request on one shared vCPU, loopback without TLS, with throughput falling from ~15,900 to 1,438–3,027 req/s; refusal is cheaper than service. Code and raw results are archived (Zenodo, MIT). Main limitations: no real-world deployment or operator-adoption evidence; benchmarks exclude TLS and client signing; motivating measurements come from interested vendors; DoS risk analyzed only qualitatively.
Review complete
Assessment at a glance
5
risk flags identified
● high 0
● medium 3
● low 2
5 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- AI crawlers have broken the crawl-for-referral bargain: automated traffic is 51–57.5% of requests and leading AI platforms fetched hundreds to roughly 70,900 pages per referred visit versus about 5 for search.
- robots.txt cannot express identity, purpose, terms, or price, relies on unverifiable client self-identification, and induces overblocking of research and archive crawlers (Longpre audit).
- terms.txt plus an origin-enforced signed exchange (Web Bot Auth, signed intent, delegation, vouchers, 402, receipts) lets any origin enforce per-path, per-purpose consent and compensation without a proxy or vendor.
- The full exchange adds 0.20–0.65 ms per request on one shared vCPU in a ~600-line dependency-free Node.js implementation, with refusal cheaper than service.
- The protocol cleanly separates enforceable (pre-delivery), auditable (post-delivery), and contractual (post-use) obligations, and cannot itself prevent training on delivered content.
Strengths
Unusually honest scope discipline
Section VI states that a signature proves a declaration was made, not that it is true, and that HTTP cannot govern post-delivery use; the paper never claims to prevent training on delivered bytes. Section IV similarly concedes the metric publisher 'also sells the remedy'.
Standards composition rather than novel cryptography
Section V composes RFC 9421 signatures, the IETF Web Bot Auth draft (WG-adopted Sept 2026), RFC 9651 structured headers, AIPREF vocabulary, and 402 negotiation; Table I fairly maps identity/purpose/terms/enforcement-locus across nine mechanisms, and the gap claim (all enforcing mechanisms are proxy-bound) is accurate as stated.
Reproducible, well-documented microbenchmark
Self-adversarial testing
Falsifiability criteria
Weaknesses
Adoption is unevidenced
Benchmark external validity
Vendor-conflicted motivating evidence
DoS analysis is qualitative
Settlement layer is assumed, not designed
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
This is a protocol design paper by an independent researcher, submitted to an IEEE Internet Computing special issue on Future Internet Systems with LLMs and Agents. It makes three claims. First (Sections I–III), assembled public measurements show the web's crawl-for-referral bargain has broken: automated traffic at 51% (Imperva 2024) to 57.5% (Cloudflare, June 2026), training rising from 72% to 79% of Cloudflare-classified AI-crawler requests (July 2024→2025), and Anthropic's crawl-to-refer ratio once near 70,900:1 versus roughly 5:1 for search. Second (Section III), robots.txt (RFC 9309) cannot express identity, purpose, terms, or price, is unverifiable, and induces overblocking (the Longpre audit of 14,000 domains). Third (Sections V–VII), terms.txt plus an origin-enforced exchange — Web Bot Auth signatures, signed Access-Intent, delegation tokens, vouchers, HTTP 402, signed hash-chained receipts — enforces per-path, per-purpose consent and compensation, implemented in ~600 lines with 0.20–0.65 ms measured overhead. The paper's distinguishing virtue is boundary discipline: Section VI states plainly that signatures prove a declaration was made, not that it is true, and that the design cannot govern post-delivery use.
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 5 strengths & 5 weaknesses, analyzed in detail
- 5 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What is terms.txt?
terms.txt is a proposed robots.txt-style file published at /.well-known/terms.txt where a website declares per-path, per-purpose machine access rules — allow, charge a price, or deny — for purposes such as search, agent, train-ai, and archive. Unlike robots.txt it is not advisory: it is backed by an HTTP exchange that verifies the client's cryptographic identity (Web Bot Auth), checks signed intent, and enforces payment or refusal at the origin server.
How much overhead does the terms.txt protocol add per request?
In the paper's loopback microbenchmark on a single shared vCPU without TLS, the signed exchange added 0.20 ms per request for identity verification alone and up to 0.65 ms for a fully charged path with voucher and signed receipt, against a 0.054 ms passthrough baseline. Throughput fell from about 15,900 to 1,438–3,027 requests per second because Ed25519 verification ran on one core. The paper notes these figures exclude TLS and client-side signing.
Can terms.txt stop AI companies from training on my content?
No, and the paper says so explicitly. Once content is delivered, HTTP cannot govern its use. terms.txt makes access terms explicit, signed, attributable, and priced; violations after delivery are handled by audit (e.g., detecting full reproduction under a reference-only declaration) and revocation of standing, while actual training prevention remains a contractual and legal matter, such as under the EU AI Act.
How is terms.txt different from Cloudflare's Pay Per Crawl?
Pay Per Crawl runs at Cloudflare's proxy, uses a vendor bot directory, and makes Cloudflare the merchant of record. terms.txt is designed to run at any origin server without a proxy, using open standards — Web Bot Auth signatures built on RFC 9421, structured intent headers, and origin-issued signed receipts — so publishers not behind a paid CDN get the same enforcement surface. The trade-off is that Cloudflare's version exists in production while terms.txt is a prototype.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Figure 1 (architecture diagram) appears only as a caption in the extracted text; the exchange flow was assessed from the Section V prose, which describes it fully.
- Tables I and II are flattened into inline text by PDF extraction; all row labels and numeric values remain legible, but column alignment was inferred.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.