Style-Debiased DPO: Updating LLM Knowledge with Factuality-Aware Synthetic Preference Data
Takayuki Yamamoto, Daisuke Kawahara
cs.CL
Quick verdict
SD-DPO inverts factually-correct rejected DPO pairs to cancel style gradients; gains on QuALITY and AToKE are well-controlled yet rest on LLM-judged labels.
Summary
The paper targets the retrieval (elicitation) side of knowledge injection: after continued pretraining (CPT) on a synthetic corpus (EntiGraph-style), knowledge is stored but not reliably retrieved as answers. Preference pairs built with gold answers as chosen and the CPT model's own samples as rejected fail under standard DPO because, when knowledge is partially known, most rejected responses are factually correct and differ only in style (about two-thirds on QuALITY, 85% on AToKE). SD-DPO uses a judge LLM (gpt-5-mini) to score factual agreement, trains factually-wrong pairs normally, and inverts style-only pairs with weight w=p/(1−p), which cancels the shared style component of the gradient at initialization (Proposition 1). On QuALITY closed-book multiple choice, SD-DPO reaches 59.17% versus a 56.21% CPT baseline (+2.96) using 10.9M preference tokens, roughly 40× fewer than the ~409M additional CPT tokens estimated for the same gain; vanilla DPO, IPO, and KTO stay at or below baseline, SamPO gains +1.11, and removing the inversion drops accuracy to 55.82%. On AToKE knowledge updating, SD-DPO attains 0.982 overall (CPT only 0.885; vanilla DPO 0.738), consistently across three model families, answering with the period-appropriate fact; parameter-editing baselines reach at most 0.303 on old-fact recall under a different protocol. General-capability loss attributable to the preference stage is −0.008/−0.013. Main limitations: all labels and evaluations are LLM-judged without human verification, AToKE uses a single seed, pronounced verbosity and repetition appear at larger training budgets (14% of generations hit the length cap at the final checkpoint), and the comparison with parameter editing is cross-protocol and reference-only.
Review complete
Assessment at a glance
4
risk flags identified
● high 0
● medium 3
● low 1
5 strengths · 5 weaknesses · 5 author questions
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- In self-generated DPO data for partially known knowledge, most rejected responses are factually correct (about two-thirds on QuALITY under cFA, ~85% on AToKE under pFA), so standard DPO suppresses knowledge that CPT just stored.
- Inverting style-only pairs with weight w=p/(1−p) cancels the style component of the expected gradient at initialization for any p∈(0,1) (Proposition 1), and this inversion — not budget tuning — produces the gains.
- On QuALITY, SD-DPO adds +2.96 points over a 56.21% CPT baseline using 10.9M preference tokens, roughly 40× fewer than the ~409M additional CPT tokens estimated for the same gain.
- On AToKE, SD-DPO reaches 0.982 overall and answers with the fact that holds at the queried period (HES 0.954), where parameter editors keep the old fact only 2-30% of the time under a different protocol.
- The preference stage preserves general capabilities (−0.008 on QuALITY, −0.013 on AToKE across six benchmarks), consistent with the KL-bounded analysis of Appendix A.2.
Strengths
Mechanism is isolated by targeted ablations
Filtering without inversion stays within seed noise (+0.24, Table 1); removing only the inversion drops SD-DPO from 59.17% to 55.82% (Table 2, dagger); all nine vanilla-DPO filter/learning-rate combinations stay ≤ +0.58 (Table 9). The gain is pinned to the inversion, not to budget tuning.
The failure mode is quantified, not just asserted
Section 4.1 and Table 4 measure the share of factually-wrong rejected responses (p≈0.33 under cFA on QuALITY, p=0.148 under pFA on AToKE); Table 12 shows rejected lengths are near-identical across the ci=1 and ci=0 groups, ruling out the split as a length proxy.
Unusually thorough and honest baseline coverage
Theory is modest, stated, and stress-tested
Cross-family validation on the main task
Weaknesses
The entire pipeline is LLM-judged with no human verification
Verbosity side effect interacts with lenient judging
Thin statistical reporting
Theory covers only the initialization point
Asymmetric preprocessing between SD-DPO and baselines
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
The paper splits knowledge injection into **storage** (CPT on synthetic data, EntiGraph-style) and **elicitation** (a preference-optimization stage). Its diagnosis: when the target knowledge is partially known, self-generated DPO pairs are dominated by factually correct rejected responses differing from gold answers only in style — measured at p≈0.33 (cFA) on QuALITY and p=0.148 (pFA) on AToKE (Table 4). Standard DPO then suppresses correct stored knowledge. SD-DPO scores factual agreement with a judge LLM, inverts style-only pairs with weight w=p/(1−p), and proves (Proposition 1) that the style component of the expected gradient cancels at initialization. Central claims: (i) +2.96 on QuALITY over a 56.21% CPT baseline using 10.9M preference tokens (~40× cheaper than the ~409M CPT tokens estimated for the same gain, Figure 2); (ii) 0.982 overall on AToKE with period-appropriate old/new-fact retrieval, consistent across three model families; (iii) near-zero general-capability loss (−0.008/−0.013, Table 10). The claims are scoped honestly; the AToKE comparison to ROME/MEMIT/METO is explicitly marked reference-only due to protocol differences.
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 5 strengths & 5 weaknesses, analyzed in detail
- 4 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What is SD-DPO and how does it differ from standard DPO?
SD-DPO (style-debiased DPO) is a variant of direct preference optimization for self-generated preference data. A judge LLM scores whether the rejected (model-generated) response is factually correct relative to the chosen (gold) answer. Factually-wrong pairs are trained as in standard DPO; pairs whose rejected response is correct and differs only in style (length, wording) are trained with the preference direction inverted and weighted by w=p/(1−p), so the style component of the gradient cancels across the dataset and only the factual difference is learned.
Why does standard DPO fail after continued pretraining on new knowledge?
When a model partially knows the target facts, most of its own sampled answers are already factually correct and differ from the gold answer mainly in style. Standard DPO pushes all rejected responses down, so it suppresses correct knowledge the CPT stage just stored. In this paper, vanilla DPO lands below the CPT-only baseline on both benchmarks (52.77% best on QuALITY vs 56.21% baseline; 0.738 overall on AToKE vs 0.885).
How much cheaper is SD-DPO than training on more synthetic CPT data?
On QuALITY, SD-DPO's +2.96-point gain over the 56.21% CPT baseline cost 10.9M preference tokens. Based on EntiGraph's scaling curve, reaching 59.17% by CPT alone would need roughly 409M additional tokens — about 40× more. The authors note this equivalence rests on a single measured point of the curve.
Does SD-DPO hurt the model's general abilities?
Reported changes over six benchmarks (MMLU, HellaSwag, ARC-C, Winogrande, TruthfulQA, GSM8K) are small: −0.071 average from the pre-CPT base to the final SD-DPO model on QuALITY, of which only −0.008 is attributable to the SD-DPO stage (−0.013 on AToKE). The clearest side effect is verbosity: outputs grow substantially and a fraction of generations repeat to the length cap, especially at later checkpoints.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Figures 1-3 are available only as extracted caption text; visual details (e.g., the scaling-curve overlay in Figure 2) could not be independently verified.
- The abstract text is truncated mid-sentence in the extraction; the body sections A-G and Tables 1-13 appear complete.
- All AToKE comparisons with ROME/MEMIT/METO use a different base model and evaluation protocol and are labeled reference-only by the authors; numbers were treated accordingly.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.