Value Over Language Model: Detecting Original Contribution in Writing
Vibhhu Sharma, Thorsten Joachims, Sarah Dean
cs.AI · cs.CL
Quick verdict
VOLM reframes AI-text detection as measuring human-added information via LLM surprisal; promising and honest, but per-document validation is thin.
Summary
The paper proposes VOLM (Value Over Language Model), a training-free framework that measures how much original content a human contributed to a document, in contrast to surface-level LLM-text detectors. VOLM extracts a document's content at k increasing granularity levels (here, schema-constrained fact lists with WordNet-canonicalized actions), prompts an LLM to reconstruct documents from each partial representation with length matching, and scores reconstructions by average per-token log-probability under the evaluator model conditioned only on the task description T. Scores are compared against a replacement-level reference document generated from T alone; the earliest granularity level with persistent significant difference (two-sided test, p<0.05) yields VOLM = 1 − i*/k. Using Llama 3.1 8B on 100 All the News articles, 200 ICLR 2023 reviews, and 200 PERSUADE 2.0 essays, human curves separate from reference curves early (news: level 0.1) and stay separated under bullet-reconstruction and German/Korean round-trip translation. Qwen-generated and DIPPER-humanized texts fall intermediate — receiving non-zero VOLM — which the authors attribute to residual stylistic leakage; increasingly constrained extractors reduce the Llama–humanized AUC gap from 0.367 to 0.101 (72.4%). Limitations: main results are corpus-level (m=n=1), the full per-document procedure is illustrated on a single document, only 8B-scale models are tested, there is no quantitative head-to-head with existing detectors, and no code is released.
Review complete
Assessment at a glance
5
risk flags identified
● high 0
● medium 3
● low 2
4 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- Existing LLM-text detectors measure surface authorship, not whose ideas a document expresses, and therefore misjudge LLM-assisted but human-originated writing (Section 1).
- VOLM — extraction at increasing granularity, LLM reconstruction, and surprisal scoring against a replacement-level document M(T) — separates human-authored from generic-prompt LLM corpora at the population level across news, ICLR reviews, and essays (Sections 4.3, A.2–A.3).
- VOLM is substantially invariant to content-preserving transformations including bullet-reconstruction, LLM rewriting, and German/Korean round-trip translation (Section 4.4, Fig 2b).
- Residual stylistic leakage causes other-model and DIPPER-humanized text to receive non-zero VOLM, and increasingly constrained extractors reduce the Llama–humanized AUC gap by 72.4% (0.367 → 0.101) without moving the Humanized–Human gap (Section 5, Fig 5).
Strengths
Genuine problem reframing
Section 1 separates 'who wrote the words' from 'who supplied the ideas' with concrete scenarios (reviewer supplying specific technical flaws; non-native author's translated argument), and Section 2 precisely differentiates VOLM from DetectGPT, Liang et al.'s corpus statistics, EditLens, Xie et al.'s φ = I(x;y)/I(y) (critiquing its unconditional-surprisal floor), and adversarial compression.
Style insulation by construction
The pipeline never scores the original document's surface text — only model-generated reconstructions, length-matched to ±10% (Section 3.1) — and Figure 4 shows concretely how word choices persist through condensation/reconstruction, motivating the design.
Transparent negative results and quantified extractor ablation
Unusually complete reproducibility inputs
Weaknesses
Per-document machinery validated on a single example
Non-zero VOLM for unoriginal, stylistically shifted text
No quantitative head-to-head with existing detectors
Underspecified statistics
Narrow model scale and unvalidated task descriptions
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
This method paper reframes LLM-writing forensics: instead of asking what fraction of surface text a model wrote (DetectGPT; Liang et al. 2024a,b; EditLens, Thai et al. 2025), it asks how much information content originated with the human author. Section 1 motivates the distinction with two concrete cases — a reviewer who supplies specific technical flaws and lets an LLM render them into prose, and a non-native speaker whose argument a model translates — both of which surface detectors would flag despite genuine human contribution. VOLM operationalizes contribution as surprisal: extract a document's content at k increasing granularity levels, have the evaluator model M reconstruct documents from each partial representation, and compare average per-token log-probabilities (conditioned only on task description T) against those of a replacement-level reference document Dref = M(T); the name borrows from baseball's VORP. Claims are stated carefully: corpus-level separation of human vs generic-prompt LLM corpora across three domains, invariance to content-preserving transforms, and an explicit admission that stylistic leakage inflates scores for other-model/humanized text.
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 4 strengths & 5 weaknesses, analyzed in detail
- 5 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What is VOLM?
VOLM (Value Over Language Model) is a training-free framework that estimates how much original content a human author contributed to a document. It extracts the document's content at increasing granularity, has a language model reconstruct documents from each partial representation, and measures how surprising those reconstructions are compared with a document the model produces from the task description alone.
How is VOLM different from AI text detectors like DetectGPT?
Surface detectors estimate what fraction of the text was machine-written, using properties of the text itself. VOLM never scores the original text's style: it extracts content, rebuilds documents from that content with the evaluator model, and asks how much information goes beyond what the model would produce from a generic prompt — so heavy LLM use that merely renders the author's own ideas does not lower the score.
Does VOLM produce false positives on humanized or other-model text?
Partially, yes. In the paper's experiments, DIPPER-humanized text and text generated by a different model (Qwen) received non-zero VOLM scores because stylistic differences leak through extraction and reconstruction. More constrained extractors reduced but did not eliminate this effect (residual AUC gap 0.101 vs the original 0.367).
What datasets and models were used in the experiments?
100 articles from All the News (task = headline), 200 ICLR 2023 reviews (task = full paper plus reviewer guidelines), and 200 PERSUADE 2.0 persuasive essays (task = essay prompt), with Llama 3.1 8B Instruct as the evaluator model, Qwen 3-8B as an alternative generator, and DIPPER as the humanizer.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Figure plots (Fig 2, 3, 5–7) are images absent from text extraction; all numeric findings cited here come from the surrounding prose and captions, which are present in full.
- The metadata abstract is cut mid-word ('and r'), but the embedded full text contains the complete abstract, so no content was lost.
- One garbled sentence in Related Work ('...work stuudying methods that could Finally, our framework...') is present in the extracted text and appears to be a paper-side editing artifact, though extraction artifacts cannot be fully ruled out.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.