Exact Feedback Is Not Control: Evaluating Text-based Closed-Loop Revision in LLMs
Haitong Jiang, Chunlin Liu, Yile Wang, Yuhong Feng
cs.CL
Quick verdict
Rigorous 19-model study: exact, complete feedback still doesn't make LLM closed-loop revision reliable; careful methodology, but tasks are toy-like.
Summary
The paper asks whether large language models can reliably revise text when given complete, deterministically verified feedback. A fixed-budget protocol (T=8 revisions) runs a request–model–verifier loop over three constraint families—exact-length with content anchors, lexical and compositional structures derived from COLLIE—evaluating 19 open and closed models over 480 cases per family, 27,360 trajectories in total. Final joint success ranges from 17.4% (Qwen3 1.7B) to 99.8% (GPT-5.6 Sol); Llama 3.1 8B reaches 75.8%, matching or beating several API systems. Fixed-draft crossovers on exact-length tasks show 42.5–77.1-point gaps persisting from identical drafts, so failures are not inherited from initial generation. Controlled one-step experiments expose model-specific feedback-to-action response curves that replicate on held-out texts (Spearman ρ>0.85); post-training and scale reshape rather than uniformly improve responses. Failed trajectories frequently repeat earlier outputs (84.8–90.2% across families), and prior recurrence predicts lower later recovery (adjusted OR≈0.05). History-reset interventions at matched recurrent states strongly change immediate escape (+42–49 points) but improve final success only modestly (+5.0–10.8 points). Limitations include toy-like deterministic word-count tasks, near-ceiling compositional results, distinct datasets per family making cross-family comparison descriptive, and potential API version drift. Overall a rigorous, unusually honest empirical evaluation.
Review complete
Assessment at a glance
4
risk flags identified
● high 0
● medium 3
● low 1
4 strengths · 4 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- Under complete, exact, deterministic feedback, mean final joint success varies enormously across 19 LLM controllers (17.4%–99.8% over three constraint families), and large gaps persist even from byte-identical initial drafts (42.5–77.1 pp).
- Models exhibit reproducible, model-specific feedback-to-action response curves (discovery–confirmation Spearman ρ>0.85); post-training and scale reshape these responses without consistently moving them toward exact correction.
- Failed trajectories frequently repeat earlier outputs (78.7–90.2% across families), and prior recurrence is associated with substantially lower subsequent recoverability (adjusted OR≈0.05).
- Removing dialogue history at matched recurrent states strongly changes immediate escape (+42.2/+49.2 pp for Llama/GLM) without reliably improving final success (+10.8/+5.0 pp), with effects depending on model, task, and trigger-state composition.
- Exact feedback provides observability, not control: reliable closed-loop revision is a property of the model–state interaction rather than of feedback quality alone.
Strengths
Clean isolation of model-side revision
Deterministic verifiers report all remaining violations and the same frozen rules both generate feedback and decide success (Section 3, Eq. 1–2), so feedback correctness/completeness is a controlled condition — a sharper design than prior self-correction evaluations (Feedback Friction, RefineBench, Stechly et al. 2025).
Fixed-draft crossover decisively decouples revision from generation
Table 2 / F.1: 240 shared exact-length cases with byte-identical initial drafts; Llama 3.1 8B keeps a 42.5–77.1 pp final-success advantage (all Holm-significant), even revising peers' drafts better than its own — revision gaps are not inherited from initial generation ability.
Scale plus rigorous, preregistered inference
Unusually honest reporting of negative results
Weaknesses
Limited ecological validity
Fragile headline aggregation
Small, heterogeneous intervention samples
Ceiling effects and drift risk in parts of the design
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
The paper asks a narrow but well-posed question: when feedback is complete, correct, and deterministically checkable, how reliably do LLMs convert it into successful revisions? It introduces a fixed-budget (T=8) request–model–verifier protocol over three constraint families — exact-length with content anchors (480 self-built cases from Dolly, No Robots, WritingPrompts, OASST1), lexical (COLLIE structures c05/c07/c12 at 240/120/120), and compositional (c09/c10 at 240/240) — and evaluates 19 controllers (12 local checkpoints, 7 API systems), yielding 27,360 trajectories. Central claims: (1) final joint success ranges from 17.4% (Qwen3 1.7B) to 99.8% (GPT-5.6 Sol, Table 1); (2) gaps persist from identical drafts (42.5–77.1 pp, Table 2); (3) feedback-to-action responses are model-specific and reproducible (Spearman ρ>0.85 discovery–confirmation); (4) recurrence marks low recoverability (adjusted OR 0.047, Table 3); (5) history resets change immediate behavior (+42.2/+49.2 pp escape) without proportionate final-success gains (+10.8/+5.0 pp, Figure 6). The titular conclusion — exact feedback provides observability, not control — is supported within its declared scope.
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 4 strengths & 4 weaknesses, analyzed in detail
- 4 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What does 'exact feedback is not control' mean in this paper?
Even when a deterministic verifier tells the model every remaining violation — e.g., current versus target word count and any missing required words — many models still fail to fix the text within eight revisions. Precisely knowing the error (observability) does not guarantee the model can translate it into a correct edit (control).
Which models were evaluated and how large are the differences?
19 controllers including Llama 3.1 8B/70B, the Qwen3 series, Gemma 2 9B, GLM-4, Granite 3.3, Falcon-H1, GPT-5.6 Sol, Claude Opus 5, Gemini 3.6 Flash, GLM-5.2, DeepSeek V4 Flash, and Qwen3.7 Plus. Mean final joint success across three constraint families ranged from 17.4% (Qwen3 1.7B) to 99.8% (GPT-5.6 Sol); Llama 3.1 8B reached 75.8%, beating several API systems.
Is the code and data available?
Yes. The paper links a public repository (github.com/kevinjiang0121-cyber/exact-feedback-code) with frozen evaluation cases, verifier and pipeline code, tests, and reproduction instructions. Frozen experimental outputs are not included, and rerunning the API panel requires user-supplied credentials.
Does deleting the conversation history help the model revise?
It changes behavior a lot but helps completion only a little. In matched-state experiments, removing earlier dialogue raised immediate escape from output repetition by 42–49 percentage points for Llama 3.1 8B and GLM-4 9B, yet final success improved by only 10.8 and 5.0 points respectively — most reset continuations still failed.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Extracted text omits parts of Section 4.1 and C.3 and truncates some Appendix F tables; Figure 4 curve shapes are inferred from captions and surrounding text rather than inspected directly.
- The API panel names 2026-era proprietary systems (GPT-5.6 Sol, Claude Opus 5, Gemini 3.6 Flash); this review cannot verify endpoint identity or versions.
- Appendix B.2 content-contract robustness checks are reported only for Llama 3.1 70B versus Qwen3.7 Plus.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.