Technical Manual for Toolkit for Confidence-Corpus Consistency via Fine-Tuning on a Fabricated Corpus

Summary

This paper documents, in the form of a technical manual, an open-source toolkit designed to test whether a language model's confidence in an answer tracks consistency with its training corpus rather than correspondence with fact. The instrument fine-tunes a small causal LM (default Qwen2.5-0.5B) on a synthetic corpus that asserts one fabricated answer for each of the 81 single-digit addition pairs, then compares post-fine-tuning confidence in each fabricated answer against pre-fine-tuning confidence in the corresponding true answer, using an identical teacher-forced, token-length-normalized log-probability procedure throughout. Design choices explicitly target confounds: an exhaustive, unambiguous fact space; per-token mean normalization that neutralizes single- versus double-digit tokenization asymmetry; a seeded, reproducible fabricated-fact draw; and a baseline-validation stage confirming genuine pre-fine-tuning arithmetic competence. The manuscript deliberately reports no empirical results and interprets no run, positioning itself purely as instrument documentation; the toolkit is archived on Zenodo with a pinned dependency environment. Main limitations, largely self-acknowledged: a single fact domain, small-model scale only, no training-run seed or significance testing, fixed hyperparameters, one fabricated value per pair, no engagement with the knowledge-editing literature, and a fine-tuning loss that also trains prompt tokens. Its scientific value depends entirely on future empirical use.