Stick to What You Know: A Study of Knowledge-Aligned Supervised Fine-Tuning
Arthur Becker, Jakob Kemmler, David Thulke, Christine Schäfer, Christian Dugast, Hermann Ney
cs.CL · cs.AI · cs.LG
Quick verdict
Proposes knowledge-aligned SFT framing and Recall Rewrite; factuality gains on WildHalu/Bios are real but partly reflect refusal-driven FActScore inflation.
Summary
The paper studies hallucinations arising when SFT targets require knowledge the base model has not internalized. It formalizes knowledge-aligned SFT — constraining training targets to the base model's parametric knowledge — and unifies existing methods (FLAME's self-generation, UNITcut's confidence filtering) plus two new variants under one data-construction algorithm: Evidence Rewrite (verifies base-model self-generations against retrieved Wikipedia evidence) and Recall Rewrite (retains gold-response claims only when 5 probing questions with 2 sampled base-model answers each consistently entail them, thresholds 2/1/2/1). Training Qwen3-4B and OLMo-3-7B on the 3,468-example English OASST1 subset, Recall Rewrite raises FActScore from 74.4 to 84.1 (WildHalu) and 34.1 to 76.4 (Bios) over standard SFT while general capabilities stay within a 2.1-point OLMES band; UnknownBench refusal F1 improves on all subtasks at lower precision. A controlled ablation (Section 4.4) varying only the share of known claims (100/50/0%) confirms a causal monotonic effect. Key limitations, largely acknowledged by the authors: FActScore counts refusals as fully supported while supported-claim counts drop (Bios 5,061 to 2,561), so gains partly reflect a more conservative response policy; pipelines rely on different proprietary teachers (gpt-4o-mini vs gpt-5-mini); evaluation is entity-centric; and the probing pipeline's scalability beyond small data is untested.
Review complete
Assessment at a glance
4
risk flags identified
● high 0
● medium 3
● low 1
5 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- Knowledge-aligned SFT — constraining SFT targets to the base model's parametric knowledge — reduces measured factual hallucinations on WildHalu and Biography while general capabilities stay within a 2.1-point OLMES band (Tables 1, 5).
- Recall Rewrite, which keeps only claims the base model consistently recalls under probing (Equation 1, thresholds 2/1/2/1), yields the strongest factuality gains and the best UnknownBench refusal F1 on all three subtasks (Tables 1, 4).
- Varying only the share of known claims in otherwise identical targets causally modulates hallucination (Table 3: WildHalu FActScore 79.5/80.4/86.1 at 0/50/100% known).
- Naive generation-based alignment (FLAME) does not improve over standard SFT (74.4 vs 74.4 WildHalu FActScore), showing self-generation is an unreliable proxy for parametric knowledge.
Strengths
Unified method formalization
Algorithm 1 and Table 7 cast FLAME, UNITcut, Evidence Rewrite and Recall Rewrite as one data-construction template differing only in GATE/SOURCE/UNKNOWN hooks, so method differences are isolated structurally rather than by incidental pipeline choices.
Causal dose-response ablation
Section 4.4 (Table 3) varies only the share of known claims (100/50/0%) with identical prompts, refusal counts (302 refusals / 1,777 answers) and hyperparameters: WildHalu FActScore rises 79.5 to 80.4 to 86.1 and Bios 38.4 to 38.7 to 69.9, reproducing Gekhman et al. (2024) in open-domain instruction data.
Unusually honest metric analysis
Quantified failure modes
Strong artifact release
Weaknesses
FActScore rewards refusal
Teacher-model confound
Single-run statistics
Over-refusal generalizes beyond the knowledge boundary
Narrow evaluation scope
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
The paper reframes SFT-induced factual hallucination as a supervision-design problem: targets demanding claims outside the base model's parametric knowledge K(Mbase) teach the model to guess. It formalizes knowledge-aligned SFT (Section 2, Figure 1), unifies FLAME (generation-based) and UNITcut (estimation-based) with two new methods — Evidence Rewrite and Recall Rewrite — under one data-construction algorithm (Algorithm 1, Table 7), and claims (i) factuality gains on WildHalu and Biography, (ii) preserved general capabilities, (iii) improved refusal calibration on UnknownBench, and (iv) a causal dose-response between the share of known claims and hallucination (Section 4.4).
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 5 strengths & 5 weaknesses, analyzed in detail
- 4 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What is knowledge-aligned SFT?
A fine-tuning approach where SFT training targets are constrained to facts the base model already knows: claims the base model cannot reliably recall are filtered or rewritten out of the supervision data. The goal is to reduce hallucinations by not pressuring the model to produce facts it never learned, rather than to inject new knowledge.
What is Recall Rewrite and how does it work?
A data-construction method introduced in this paper. Each factual claim in a gold SFT response is probed with 5 independently generated questions; the base model answers each question twice; a claim counts as 'known' only if at least 2 questions get an entailing answer and at most 2 get a contradicting one (thresholds 2/1/2/1). Claims classified as unknown are removed and the response is rewritten, or turned into a refusal if too little remains.
How much does Recall Rewrite improve factuality?
On Qwen3-4B trained on the OASST1 subset, FActScore rises from 74.4 to 84.1 on WildHalu and from 34.1 to 76.4 on Biography versus standard SFT. However, the model also refuses far more often (252 vs 4 refusals on Bios) and produces fewer supported claims in total (5,061 to 2,561 on Bios), so part of the gain is a more conservative response policy.
Does knowledge-aligned SFT hurt general capabilities?
Not clearly in this study. On the OLMES average (HumanEval+, GSM8K, IFEval, TruthfulQA), the Recall Rewrite model scored 68.9 vs 69.8 for standard SFT — a 0.9-point gap within the 2.1-point band spanned by all compared models. The IFEval difference is traced to 30 refusals, mostly on creative-writing prompts; on non-refused prompts the models are on par (56.3 vs 56.5).
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- The provided text is a sampled excerpt with marked '[...sampling gap...]' sections (part of Section 4.1 including Table 10 API costs, portions of the References and appendices, Examples 10-11 in Section I.2, part of Section J); all numbers cited in this review come from visible text.
- The content of Footnote 3 (the data-release pointer) was not in the visible text, so the exact artifact location and whether code (beyond data) is released could not be verified.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.