Do LLMs Make More Mistakes If They Do Not Believe the Input Data?
Peter Kochelka, Aleš Manuel Papáček, Vojtěch Dvořák, Ondřej Dušek
cs.CL
Quick verdict
Finds weak context-memory conflict (-0.05/5 faithfulness); judge-audit design is strong, but small human samples and ceiling effects temper the null claim.
Summary
The paper asks whether large language models are less faithful to input data they perceive as implausible (context-memory conflict). The authors build 2,847 RDF triples from Czech and Slovak local knowledge (CUS-QA) in factual, counterfactual, and fictional variants, prompt nine open-weight generators (1.7B-120B) to verbalize them in English, Czech, Slovak, and Upper Sorbian (11,388 generations per model, ~102,492 outputs), and score faithfulness and fluency with LLM judges audited against 648 author-annotated samples. Kimi K3, the primary judge (quadratic-weighted kappa 0.71 on heldout data), shows a counterfactual faithfulness penalty of only -0.049 on the 1-5 scale (fictional: -0.030); confidence intervals exclude zero but the effect is negligible. The same outputs scored by DeepSeek V4 Pro Preview yield a -0.177 penalty, three to four times larger, despite acceptable aggregate agreement - demonstrating that a judge can carry condition-aligned bias undetected. Larger models classify input factuality better (33-36% for small models vs ~54% for the largest) and generate more faithfully; small models collapse in Upper Sorbian (mean 1.43). Explicit instructions to treat conflicting triples as authoritative change scores by only -0.012. Main limitations: single-author annotation without adjudication, no Upper Sorbian speaker among annotators, LLM-constructed data without reported manual verification, reasoning disabled, and near-ceiling mean scores (~4.5) that may compress measurable effects.
Review complete
Assessment at a glance
4
risk flags identified
● high 0
● medium 3
● low 1
4 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- Counterfactual inputs incur only a -0.049 faithfulness penalty (1-5 scale) under the human-validated Kimi K3 judge, indicating a weak context-memory conflict in data-to-text generation from local-knowledge RDF triples.
- Judge choice materially changes the measured effect: DeepSeek V4 Pro Preview reports -0.177 on identical outputs despite aggregate agreement (kappa_w 0.555) that would not flag its condition-aligned bias.
- Larger generators classify input factuality more accurately (33-36% for small models vs 52-54% for large, chance ~33%) and generate more faithfully, with all models weakest in Upper Sorbian (small models: mean 1.43).
- Explicitly instructing generators to treat conflicting triples as authoritative barely changes scores (pooled -0.012, Table 19).
- Part of any measured conflict is evaluator bias: the Kimi-human gap shrinks from +0.35 on factual to +0.16 on counterfactual items across the 648 human-annotated rows.
Strengths
Rigorous, multi-layer judge audit
Five candidate judges compared against 540 tuning + 108 frozen-prompt heldout human annotations (Table 1); Kimi K3's kappa stability (0.712 to 0.711) vs DeepSeek's collapse (0.555 to 0.350) cleanly demonstrates overfitting; the disclosure ablation uses replicate-run noise floors (Table 11: Kimi +0.050 within its floor, DeepSeek +0.148 exceeding it).
Valuable negative result with quantified evaluator bias
On identical outputs, DeepSeek measures a -0.177 CFA faithfulness penalty vs Kimi's -0.049 (Section 5.3), while its aggregate agreement (kappa_w 0.555, 80.9% within one point) would not flag the problem - a concrete, transferable lesson that judge bias aligned with the experimental condition masquerades as a generator property.
Large matched-design experiment spanning a severely low-resource language
Consistent robustness probing
Weaknesses
Thin, unadjudicated human annotation
Tuning asymmetry undermines the judge ranking
Ceiling compression weakens the null claim
Fully LLM-mediated data construction without reported verification
Restricted generator pool
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
The paper investigates the context-memory conflict (Xu et al., 2024): whether an LLM's perceived implausibility of input data degrades its faithfulness when verbalizing it. From Czech/Slovak local knowledge in CUS-QA, the authors construct 2,847 RDF instances in factual (FA), counterfactual (CFA, same-type entity substitution), and fictional (FI) variants, and have nine open-weight generators (Qwen3.5 122B-A10B, gpt-oss-120b down to Qwen3 1.7B) verbalize them in English, Czech, Slovak, and Upper Sorbian - 11,388 generations per model, ~102,492 outputs total (Section 4.1). Central claims: (1) the conflict is weak - CFA costs only -0.049 faithfulness on the 1-5 scale under the primary Kimi K3 judge, FI -0.030 (Table 4); (2) judge choice changes the story - DeepSeek V4 Pro Preview measures -0.177 on the identical outputs; (3) larger models are more faithful and better factuality classifiers; (4) part of the FA>FI>CFA ordering is evaluator bias, since the Kimi-human gap shrinks from +0.35 (FA) to +0.16 (CFA).
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 4 strengths & 5 weaknesses, analyzed in detail
- 4 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What is the context-memory conflict in LLMs?
It is the tension that arises when a model's internal (parametric) knowledge disagrees with information supplied in the prompt: the model may follow the context, or silently correct, hedge, or ignore it in favor of what it believes. This paper measures whether that conflict makes models less faithful when turning structured data (RDF triples about Czech and Slovak local knowledge) into natural-language text.
Do LLMs ignore counterfactual information they are given?
In this study, only marginally. Generating text from counterfactual rather than factual triples cost just 0.049 points of faithfulness on a 1-5 scale under a human-validated judge. A few documented cases exist - e.g., a model restoring the real 1620 date for the Battle of White Mountain when given 2013 - showing models sometimes override input facts, but the aggregate effect is tiny.
Can LLM-as-a-judge evaluation results be trusted?
Yes, but only after auditing against human labels, and even then with care. The paper's central methodological lesson is that a judge can agree well with humans overall (quadratic-weighted kappa 0.555, ~81% of scores within one point) while carrying a bias tied to your experimental condition: on identical outputs, one judge measured a counterfactual faithfulness penalty three to four times larger than another judge.
Which languages and models were tested in this paper?
Generation was tested in English, Czech, Slovak, and Upper Sorbian - a West Slavic language with roughly 25,000 speakers - with input data drawn from Czech and Slovak local knowledge unlikely to be fully in training corpora. Nine open-weight generator models were used, from Qwen3 1.7B up to Qwen3.5 122B and gpt-oss-120b, all with reasoning disabled or set to low.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Figures 1 and 2 (data-pipeline diagram, paired-score matrix) are image elements whose visual content is not recoverable from text extraction; their captions and in-text descriptions supply the needed information, and no conclusions depend on the missing visuals.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.