Geometry of Values: Task Vector Composition for Ethical Preference Alignment in Language Models
Utkarsh Agarwal, Monojit Choudhury
cs.CL · cs.AI
Quick verdict
Neat task-vector demo: LLM value stances reversible without retraining; but n=60 gold set, no significance tests, and keyword-leakage risk weaken the claims.
Summary
This workshop paper (Pluralistic Alignment @ ICML 2026) asks whether an ethical value preference can be learned, then isolated and reversed in weight space without retraining. The authors build a 12,000-instance dataset of two-option dilemmas over three value pairs (Honesty vs Justice, Justice vs Autonomy, Autonomy vs Honesty), generated with GPT-4o from 20 hand-written scenario domains, edited by a Gemini validator (4,055 of 12,000 samples), and machine-translated into Hindi, Arabic, Spanish, and Chinese; a 60-dilemma human-written gold set checks generalization. GPT-5-mini shows stable cross-lingual biases (~70% toward Justice-over-Autonomy and Honesty-over-Autonomy) that prompt steering only partly corrects, especially in Hindi. LoRA SFT and DPO on Llama-3.2-1B/3B eliminate position bias and reach ≥98% accuracy on the synthetic test set and ≥90% on the gold set. The core contribution is Algorithm 1: a task vector whose instruction-following component (estimated from a pair of opposing-preference checkpoints) is projected out, leaving a preference vector that, subtracted with tuned weights γ1/γ2, flips a model's stance while retaining ≥80% (1B) and mostly ≥96% (3B) of fine-tuned accuracy. Transitive composition fails — pairwise preference vectors exceed 80° apart — supporting only local modularity. Main limitations: the gold set yields ~4 examples per task-language cell, no significance testing, possible lexical leakage of value words into dilemma texts, and a future-tense code release.
Review complete
Assessment at a glance
5
risk flags identified
● high 0
● medium 4
● low 1
4 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- GPT-5-mini exhibits stable, largely language-independent value biases (~70% toward Justice-over-Autonomy and Honesty-over-Autonomy) that in-context stance prompting only partially overrides, worst in Hindi (Tables 1–2).
- LoRA SFT and DPO remove first-option bias in Llama-3.2-1B/3B and reach ≥98% accuracy on the synthetic test set and ≥90% on the 60-item human gold set (§5.2, Figs 2–3, 5–6).
- A preference vector — obtained by projecting a stance fine-tune's task vector onto the orthogonal complement of an estimated instruction-following direction — can be subtracted to reverse the value stance without retraining, retaining ≥80% (1B) and mostly ≥96% (3B) of fine-tuned accuracy (Algorithm 1, §5.3).
- Pairwise preference vectors are near-orthogonal (>80° apart), so value stances are locally reversible but do not compose transitively under linear task arithmetic (§6).
Strengths
Well-motivated decomposition with an honest failed baseline
Naive reversal of the full task vector produced garbage outputs (§4.4), reported transparently; this motivates Algorithm 1's split of the delta into an instruction-following direction (estimated as ½(Δ_S2 + Δ_Ṡ2) from an opposing-preference pair) and an orthogonalized preference vector — a clean, reusable application of Ilharco et al. (2023) and Gargiulo et al. (2025) projection ideas.
Candid negative result with a measured quantity
Transitive composition fails and the paper quantifies why: cosine angles between pairwise preference vectors exceed 80° (§6), correctly scoped as 'limited modularity' rather than buried — rare and valuable candor.
Genuine robustness checks on the core method
Dual training objectives and multilingual parallel design
Weaknesses
Human gold set is statistically underpowered
No statistical inference anywhere
Untested lexical leakage of value words
Quality control is almost entirely automated
Resilience claim strained by Fig 8
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
A workshop paper (Pluralistic Alignment @ ICML 2026) making four claims: (1) a formalization of ethical dilemmas (§2, Definitions 2.1–2.2) and a 12,000-instance multilingual dataset over three value pairs — A=Honesty, B=Justice, C=Autonomy, tasks AB/BC/CA (§3); (2) compact Llama-3.2-1B/3B models learn stable value policies via LoRA SFT/DPO, ≥98% on the synthetic test and ≥90% on a human gold set (§5.2); (3) Algorithm 1 isolates and reverses a value preference in weight space without retraining (§4.4, §5.3); (4) transitive composition fails — preference vectors exceed 80° apart — so only 'limited modularity' holds (§6). GPT-5-mini baselines (Tables 1–2) document inherent biases (~70% Justice≻Autonomy, ~70% Honesty≻Autonomy across five languages) and brittle prompt steering, worst in Hindi (AB: 64.5% baseline falls to 61.8% under same-direction steering).
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 4 strengths & 5 weaknesses, analyzed in detail
- 5 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What is a task vector in this paper?
A task vector is the weight difference Δ = θ_finetuned − θ_base between a base model and a fine-tuned checkpoint (Ilharco et al., 2023). This paper splits such a vector from an ethics-stance fine-tune into an 'instruction-following' component (estimated as the average of task vectors from two models trained on opposite stances of a different value pair) and a residual 'preference' component, then adds/subtracts the components with tuned weights γ1/γ2 to flip a model's value stance without retraining.
Did fine-tuning teach the models ethics, or just answer formatting?
The paper argues both components exist: the full task vector mostly encodes instruction-following (naively reversing it produced garbage output), while the orthogonalized residual carries the value preference — reversing only that residual retains ~80–100% of fine-tuned accuracy. However, the dilemmas often name the values explicitly, keyword shortcuts cannot be fully excluded, and the 60-item human gold set is too small to settle the question definitively.
What values and languages does the dataset cover?
12,000 two-option dilemmas over three value pairs — Honesty vs Justice, Justice vs Autonomy, Autonomy vs Honesty — generated with GPT-4o, validated and edited by Gemini (4,055 of 12,000 samples edited), and machine-translated into Hindi, Arabic, Spanish, and Chinese alongside English. Splits are parallel across languages (3,200 train / 400 dev / 400 test per stance), and a separately human-written gold set of 60 dilemmas exists.
Can you combine value preferences, e.g., Honesty over Justice plus Justice over Autonomy to get Honesty over Autonomy?
No — the paper reports this as a negative result. Task vectors for different value-pair preferences are nearly orthogonal in weight space (angles above 80°), and models built by chaining them performed at random after extensive coefficient search. The method supports reversing one learned binary stance at a time, not composing a full value ordering.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Heatmap figures (Figs 2–4, 5–8) were extracted as flattened number grids; the mapping of values to task/language cells is inferred and some cell attributions (e.g., the 62%/71% retention readings) may be misassigned.
- Appendix C states a public repository 'is available' but describes contents in the future tense; actual release status is unverified.
- The numeric cross-check of §5.3's '85% on the third task' against Fig 8 relies on the garbled figure text; minor misattribution risk.
- Review is based solely on the embedded extracted text; no external verification of citations or the repository was performed.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.