What Else Needs Fixing? Exploring Cost-Effective Test-Time Compute for Revision Propagation in Artifacts Generated Through Conversation
cs.CL · cs.LG
Quick verdict
New benchmark for LLM revision propagation in conversational JSON artifacts; solid cost-aware evaluation, but small synthetic data with single-annotator gold.
Summary
The paper introduces RevPropBench, a benchmark for revision propagation in JSON artifacts generated through multi-turn LLM conversation: given the conversation history, the current artifact, and a single local revision request, a model must emit an RFC 6902 patch that both applies the request and updates all dependent elements. The benchmark contains 150 samples (30 development, 120 test) from 50 human-designed scenarios across nine domains and three artifact sizes (10/50/100 elements); conversations are synthesized by GPT-5.5, tentative gold patches come from Claude-Opus-4.8, and a single human annotator spent 32+ hours correcting 23 of 150 samples. Nine methods - J/H/J+H baselines, sequential reflection (REFLECT), rule-based parallel merging (OR/AND/MAJ), medoid selection (MED), and LLM-based selection (SELECT) - are evaluated on six LLMs with five seeds. Single-inference (J+H) completion rates span 68.3-93.0%; SELECT improves most consistently (+3.3-12.5 points), MED follows (+1.8-7.7), AND degrades badly, and misses dominate errors. Cost and latency analyses recommend SELECT with four calls, or MED with three when latency matters. Main limitations: a small, deterministic-by-design dataset with one annotator and no inter-annotator agreement, no statistical significance testing, and no baselines that explicitly extract dependencies from the conversation history.
Review complete
Assessment at a glance
5
risk flags identified
● high 0
● medium 2
● low 3
5 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- Across all six models the baselines order J < H < J+H (e.g., 81.7 < 86.7 < 90.3 for qwen3.5-122b), showing that dependency-relevant context resides in the conversation history.
- Single-inference (J+H) completion rates span 68.3-93.0% across the six evaluated LLMs, scaling with model size.
- Selecting among parallel samples - via LLM judgment (SELECT, +3.3-12.5 points) or medoid (MED, +1.8-7.7 points) - is the most cost-effective test-time compute; the abstract summarizes this as a 2.2-9.7% improvement from three-sample selection.
- Unanimity merging (AND) degrades accuracy on every model, and omitted necessary edits (misses) dominate the failure modes of most methods.
- The recommended operating points are SELECT with four LLM calls, or MED with three calls when latency matters (MED k=3 adds only 1.12-1.36x latency).
Strengths
Cleanly novel problem setting
Sections 1 and 5 differentiate the setting from repository-level coding, knowledge-editing ripple effects (RippleEdits, ChainEdit), and document editing (EditPropBench, LEDGER): here dependencies are implicit and established in the conversation history, outside the artifact. The RFC 6902 patch formulation (Sec. 2.1) makes evaluation automatable and unambiguous.
Multi-stage benchmark construction with human oversight
Sec. 2.2-2.3 and App. C.2: human-designed scenarios (six propagation patterns, Table 3), GPT-5.5 conversation synthesis under an exactly-one-correct-value rule (Fig. 9), Claude-Opus-4.8 tentative golds, then a dedicated GUI annotation pass - one annotator, 32+ hours over 10 days, 23/150 samples (15%) corrected. Matchers (keyword/regex/ref_equal) and an optional-element flag handle value phrasing and legitimate ambiguity.
Evaluation breadth with a practical cost/latency axis
Diagnostic failure decomposition
Honest, specific limitations and released tooling
Weaknesses
Small test set with no inferential statistics
Single annotator, LLM-assisted gold
Determinism constraint sidesteps the hard part of the task
Missing dependency-elicitation baseline
Numeric bookkeeping inconsistencies and ceiling effects
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
The paper defines a new task: **revision propagation for conversationally generated JSON artifacts**. A user asks for one local change (e.g., "remove destination B"); the model must emit an RFC 6902 patch that applies it *and* fixes every dependent element (schedule end, per-person budget), where dependencies may be stated only in the conversation history, outside the artifact. Deliverables: (1) RevPropBench - 150 samples (30 dev / 120 test) from 50 scenarios across nine domains (travel, invoices, BOMs, access plans, ...) at three artifact sizes (10/50/100 elements); (2) an evaluation of nine methods - baselines J/H/J+H, sequential reflection (REFLECT), rule-based parallel merging (OR/AND/MAJ), medoid (MED), LLM-based selection (SELECT) - on six LLMs (gpt-oss-20b/120b, gpt-5.4-mini, qwen3.5-9b/27b/122b); (3) cost/latency guidance. Headline claims: history helps (J < H < J+H on every model); J+H spans 68.3-93.0%; selection over parallel samples is the most cost-effective test-time compute (abstract: +2.2-9.7%).
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 5 strengths & 5 weaknesses, analyzed in detail
- 5 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What is revision propagation in LLM-generated artifacts?
It is the problem of applying a user's small, local edit (for example, removing one stop from a trip plan) and automatically updating every other part that depends on it - dates, totals, budgets - so the whole artifact stays consistent. This paper tests how well LLMs do this for JSON artifacts built through multi-turn conversation, where the dependency information may exist only in the chat history, not in the artifact itself.
What is RevPropBench and how big is it?
RevPropBench is a human-annotated benchmark of 150 samples (30 development, 120 test) drawn from 50 scenarios in nine domains such as travel itineraries, invoices, and shopping carts, with artifacts of 10, 50, or 100 JSON elements. Each sample provides a conversation history, one local revision request, and gold RFC 6902 JSON patches; models are scored on whether the patched artifact exactly matches the gold result.
Which method gives the best accuracy for its cost?
The paper finds that sampling several candidate patches and selecting among them beats both single inference and reflection per unit cost: LLM-based selection with four calls (three samples plus one selection) is the recommended default, and medoid selection with three parallel calls is the low-latency alternative, adding only 1.12-1.36x latency over a single call. Naive majority voting or unanimity merging is unreliable, and strict unanimity (AND) hurts badly.
Is the code and dataset available?
Yes. The paper states that the benchmark instances, the data sampling and annotation tools, and the evaluation code are released at github.com/ntt-dkiku/llm-revision-propagation, and the appendix publishes all generation and evaluation prompts, seeds, and hyperparameters.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Figure-internal data survive the extraction only partially: Fig. 4 error-bar magnitudes, Fig. 5 exact failure counts, and Fig. 6 curve points appear as scattered axis labels; all headline numbers cited in this review are taken from the prose text.
- The printed AND range (-13.2-21.3%) lacks a minus sign on its second bound; this review reads it as 13.2-21.3 points below J+H, which is consistent with the Fig. 4 values (e.g., 47.0 vs 68.3 on qwen3.5-9b).
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.