Directing large language models to follow the letter or spirit of the law
Peng Qian, Andrew Li, Sam Chen, Sonia K. Murthy, Yonatan Belinkov, Tomer D. Ullman
cs.CL
Quick verdict
Anchored-LoRA adapters steer LLMs to letter- or spirit-based judgment across vignettes, Reddit, caselaw; solid evidence, letter-side effects weaker.
Summary
The paper asks whether large language models can be made to judge rules by the letter or by the spirit, and what internal organization underlies that distinction. The authors introduce AncLoRA, a low-rank adaptation applied to a single transformer layer (the MLP up-projection at layer 32 of Qwen3.5-27B; layer 26 of Llama-3.1-70B-Instruct), trained on 57 hand-crafted everyday-rule vignettes. The objective combines a main loss toward letter- or spirit-consistent Yes/No verdicts with an anchoring loss that preserves base-model answers on factual control questions. Adapted models shifted judgments on held-out vignettes, on 324 interpersonal 'loophole' vignettes from prior human-subjects work, on a novel 10,000-post dataset from r/MaliciousCompliance, and — in a pre-registered study — on 105 historical caselaw summaries answered in free form, while control-question judgments and MMLU accuracy stayed unchanged. Contrastive activation addition failed to steer the same distinction, supporting the authors' account of a nonlinear conceptual space; post-adaptation residual streams formed a three-dimensional manifold (Decision, Agreement, Approach) whose representational similarity (ρ = 0.76) best matched a pre-specified jurisprudential framework. Main limitations: the letter direction's effect on overinclusion was not significant in the primary held-out test; Llama showed targeting bleed into noncompliance and a nonsignificant spirit effect for caselaw; free-form caselaw answers were extracted by the same Qwen model with inference-time amplification (η = 3.5); and code availability is promised but unverified.
Review complete
Assessment at a glance
4
risk flags identified
● high 0
● medium 3
● low 1
5 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- Layer-specific AncLoRA adapters trained on 57 hand-crafted vignettes steer LLM judgments toward letter- or spirit-based interpretations without degrading factual understanding or MMLU performance.
- The steering generalizes out-of-distribution to interpersonal loophole scenarios, 10,000 real Reddit malicious-compliance posts, and 105 historical legal cases judged in free form (pre-registered).
- Contrastive activation addition fails to steer this distinction, consistent with the authors' proposal that letter/spirit concepts are organized nonlinearly in activation space.
- After adaptation, residual-stream representations form a three-dimensional manifold (Decision, Agreement, Approach) that quantitatively best matches a pre-specified jurisprudential framework (RSA ρ = 0.76).
Strengths
Progressive out-of-distribution evaluation design
Four tiers — 15 held-out rules in a novel prompt style (Fig. 1C), 324 loophole vignettes from prior human-subjects work (Fig. 2A), 10,000 noisy Reddit posts (Fig. 2B), and 105 pre-registered caselaw summaries with free-form responses (Fig. 2C) — all show directional steering, and effects persist with reasoning chains enabled (Fig. S6).
The anchoring loss demonstrably preserves factual competence
Control-question judgments show no significant deviation for either adapter direction (Fig. S2; all 95% CIs include 0), and MMLU is unchanged (Table S2: Qwen3.5-27B 88.88 base vs 88.93 spirit / 89.06 letter; Table S1: Llama 85.59 vs 85.61 / 85.56), substantiating the targeted-intervention claim.
Negative controls triangulate the mechanism
Pre-registered legal-domain study with honest null reporting
Cross-model replication and full methodological transparency
Weaknesses
Letter-direction asymmetry on the primary test
Targeting bleeds into noncompliance on Llama
Annotator circularity and inference-time amplification in the caselaw pipeline
Latent-geometry analysis is post hoc relative to the documented preregistration
Unlabeled noisy Reddit corpus; code availability unverified
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
This is an empirical interpretability-and-alignment study bridging NLP, cognitive science, and jurisprudence. It advances four claims: (1) Anchored-LoRA (AncLoRA) adapters, trained only on 57 hand-crafted everyday-rule vignettes, steer LLM judgment toward letter- or spirit-based interpretation; (2) the steering generalizes across held-out vignettes (15 unseen rules), human-subjects loophole stimuli (324 vignettes), 10,000 r/MaliciousCompliance posts, and 105 pre-registered historical caselaw summaries judged in free form; (3) the intervention is targeted — factual control questions and MMLU are unaffected; (4) post-adaptation activations reveal a three-axis conceptual space (Decision, Agreement, Approach) matching a pre-specified framework, which also explains why linear CAA steering fails. The full main text and Supplementary Materials (Figs. S1–S12, Tables S1–S2) were available for this review; figures were judged from captions and reported statistics.
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 5 strengths & 5 weaknesses, analyzed in detail
- 4 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What is the difference between following the letter and the spirit of the law in this paper?
The paper uses four behavior types: compliance (both letter and spirit kept), underinclusion (letter kept, spirit broken — e.g., a dog on a 100-foot leash), overinclusion (letter broken, spirit kept — e.g., carrying a small unleashed puppy in your arms), and noncompliance (both broken). A letter-oriented judge treats overinclusion as a violation; a spirit-oriented judge treats underinclusion as the violation.
What is AncLoRA and how does it work?
AncLoRA (Anchored Low-Rank Adaptation) is a LoRA adapter attached to one layer's MLP of a frozen LLM and trained so the model answers rule-violation questions in a letter- or spirit-consistent way. An added anchoring loss (weight λ = 0.1) keeps the model's answers on factual control questions close to the base model's, so only the interpretive stance shifts, not factual understanding.
Which models were used and did the steering hurt general ability?
Experiments used Qwen3.5-27B (adapter at layer 32) as the main model and Llama-3.1-70B-Instruct (layer 26) for replication. General ability was preserved: MMLU accuracy was essentially unchanged (e.g., Qwen3.5-27B 88.88% base vs 89.06% with the letter adapter) and answers to factual control questions did not shift significantly.
Is the code and data available?
The paper states that stimuli, code, and experimental materials will be released at github.com/pqian11/letter-spirit-adapters; the loophole stimuli come from a public OSF repository, and legal cases derive from the public Caselaw Access Project. Availability of the authors' own release could not be verified from the paper text.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Figures (Figs. 1–3, S1–S12) were available only as extracted text; visual geometry claims (Fig. 3B–C) were assessed from captions and the reported statistics.
- The Fig. 1 caption in the extraction contains a garbled parenthetical (Underinclusion and Overinclusion share one description); body-text definitions are correct, so this appears to be an extraction artifact.
- The paper states code and materials 'will be available'; the repository's current contents could not be verified from the text.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.