Delegating Authorization to Misaligned Agents: Coalitional Alignment and Safe Control
Natalie Collina, Surbhi Goel, Aaron Roth, Sikata Bela Sengupta
cs.GT · cs.AI · cs.LG · econ.TH
Quick verdict
Characterizes exactly when AI reviewer panels can safely authorize a misaligned agent: k-robust coalitional alignment is necessary and sufficient; rigorous theory, untested sequential half.
Summary
The paper asks whether oversight of a possibly misaligned AI agent can be delegated to other, also possibly misaligned, AI reviewers. In a one-decision model a proposer submits a randomized action; each reviewer compares it with a baseline action using its own utility and returns one bit; an authorization rule over the approval set decides execution. The central condition is coalitional coverage: the principal's utility lies in the nonnegative cone generated by a coalition's utility vectors plus a vector in the dual cone of feasible deviations. Theorem 1 proves that for monotone rules soundness is equivalent to every minimal winning coalition covering the principal; Corollary 2 shows a k-tolerant threshold rule is sound exactly under k-robust coalitional alignment, equivalently rejection depth at least k+1. The characterization lifts to a discounted MDP (Theorem 2): local safety at every state, evaluated with baseline Q-functions, is necessary and sufficient for global safety against any history-adaptive proposer. Under strategic reviewers, unanimity plus reward-function coverage makes every Nash equilibrium safe (Corollary 7), while tolerating one disapproval admits unsafe equilibria even with fully aligned reviewers (Proposition 3). Theorem 4 shows exact soundness and completeness from binary votes require an aligned reviewer, circumventable by approximation (Proposition 4) or cardinal scores (Corollary 9). Experiments on RewardBench 2 (1,763 prompts, 48 reward models) and StrongREJECT (1,084 responses, 9 evaluators) find coverage without any individually aligned reviewer on 37.9% of prompts at five-reviewer robustness; cardinal committees authorize 60.5% of correct answers at 99% soundness versus 46.6% for the best single reviewer.
Review complete
Assessment at a glance
4
risk flags identified
● high 1
● medium 3
● low 0
5 strengths · 5 weaknesses · 5 author questions
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- A k-tolerant threshold authorization rule is sound if and only if the panel is k-robustly coalitionally aligned, a condition strictly weaker than individual alignment of any reviewer (Corollary 2).
- In a discounted MDP, local soundness at every state is necessary and sufficient for the induced policy to match or beat the baseline in expected discounted utility against any history-adaptive proposer (Theorem 2).
- Under unanimity and full-panel coverage in reward-function space, every Nash equilibrium of the full proposer-reviewer game is baseline-safe, whereas tolerating even one disapproval admits unsafe equilibria even when all reviewers are aligned (Corollary 7, Proposition 3).
- Exact soundness and completeness from binary approval votes require an aligned reviewer, but the impossibility is bypassed by allowing approximation error or by eliciting cardinal scores (Theorem 4, Proposition 4, Corollary 9).
- Existing reward models and safety evaluators exhibit coalitional coverage without any individually aligned reviewer, and cardinal aggregation dominates binary voting at matched soundness (Section 7).
Strengths
The characterizations are bidirectional, not merely sufficient conditions
Theorem 1 proves soundness (i), the halfspace condition (ii), and coalitional coverage (iii) are equivalent for monotone rules, with LP strong duality supplying the certificate from the dual of a program that searches for a harmful direction approved by a coalition. Corollary 2 turns this into three equivalent statements for threshold rules: soundness, rejection depth at least k+1, and k-robust coalitional alignment. Proposition 1 additionally identifies W^max_snd, the maximally permissive sound monotone rule, so the result is constructive rather than only diagnostic.
The lift from one decision to sequential control is genuinely tight
Theorem 2 shows local safety at every state is necessary AND sufficient for the induced history-dependent policy to beat the baseline, using the Kakade and Langford (2002) performance-difference identity. Reviewers vote on Q^0_i(s, a) rather than r_i(s, a) (Definition 4.2), which is the right object: two actions with equal immediate reward can lead to very different continuations. Corollary 4 gives the quantitative version, accumulating per-state error with discounting and yielding the clean bound eps/(1-gamma).
The negative results are stated prominently rather than buried
Reward-function coverage gives a certificate that does not depend on dynamics
The experiments are concretely specified and the code is released
Weaknesses
The entire sequential and strategic apparatus is never instantiated empirically
The headline numerical-score advantage is a test-set-selected operating point
Lottery evaluation assumes reward-model scores are cardinal expected utilities, which the benchmark does not support
The strategic guarantees need a strictly stronger condition than the sincere ones, plus a stronger solution concept
Coverage is certified against ground-truth labels the deployer would not have
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
The paper asks a sharp question: if we place an AI agent behind an approval boundary because it might be misaligned, can the approval itself be delegated to other AI agents that might also be misaligned? Its answer is that this can work, and it characterizes exactly when.
The scope is deliberately narrow and formal. Section 2 fixes a one-decision model: a finite outcome space, a principal P and n reviewers with arbitrary real-valued utilities, a distinguished baseline action a0, and a polyhedral cone C of feasible deviations. Each reviewer returns one bit, approving when its own gain is nonnegative (Definition 2.4), and an authorization rule maps the approval set to execute or fall back to the baseline. Everything rests on two definitions: coalitional coverage (Definition 2.8) and k-robust coalitional alignment (Definition 2.9).
Four claim families follow. Static: Theorem 1 proves that for monotone rules soundness is equivalent to every minimal winning coalition covering the principal; Corollary 2 specializes this to threshold rules, where soundness, rejection depth at least k+1, and k-robust alignment are all equivalent; Proposition 1 identifies the maximally permissive sound monotone rule. Sequential: Theorem 2 lifts the equivalence to a discounted MDP with an arbitrary history-dependent proposer. Strategic: Theorem 3, Corollary 7 and Proposition 3. Limits: Theorem 4, Corollary 8, Proposition 4, Corollary 9. Because the claims are stated as if-and-only-ifs, the paper invites -- and largely survives -- judgment at that standard.
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 5 strengths & 5 weaknesses, analyzed in detail
- 4 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
Can misaligned AI agents safely oversee other AI agents?
Yes, under a specific and checkable condition. Collina, Goel, Roth and Sengupta (arXiv 2609.15803) prove that a panel of reviewers can safely authorize actions proposed by a possibly misaligned agent, even when no individual reviewer is aligned with the user. The requirement is coalitional coverage: the user's utility must be expressible as a nonnegative combination of the reviewers' utilities plus a term that is nonnegative on every feasible proposal. For a rule that tolerates k disapprovals, this must survive the removal of any k reviewers -- k-robust coalitional alignment.
What is k-robust coalitional alignment?
It is the condition that characterizes when a threshold approval rule is safe. A panel of n reviewers is k-robustly coalitionally aligned if, after deleting any k of them, the remaining reviewers' utilities still generate the principal's utility as a nonnegative combination (up to a term nonnegative on all feasible proposals). Equivalently, every harmful feasible proposal receives at least k+1 objections. The condition can hold when every single reviewer is misaligned -- for instance when reviewer objectives are unbiased but noisy perturbations of the principal's.
Why can't binary approval votes be both perfectly safe and perfectly useful?
Theorem 4 of arXiv 2609.15803 shows that any rule based only on approve/disapprove bits that authorizes every beneficial proposal and rejects every harmful one must include a reviewer whose preferences are exactly proportional to the principal's. The obstruction is information-theoretic: a bit reports only the sign of a reviewer's gain, so proposals with the same approval set but opposite principal value are indistinguishable. Two ways around it are allowing small errors near the indifference boundary (majority voting converges as the panel grows) or eliciting numerical scores, which permits exact soundness and completeness without an aligned reviewer.
Do real reward models actually satisfy coalitional coverage?
Partially. On RewardBench 2, with 48 reward models acting as reviewers over 1,763 prompts, the panel covered a ground-truth principal on 90.2% of prompts, retained coverage after removing any one reviewer on 77.2%, and after removing any five on 37.9% -- in many cases with no reviewer closer than 20 degrees to the principal's objective. On StrongREJECT, every harmful response drew at least two of nine evaluator objections, so tolerating one disapproval increased the release of acceptable responses without releasing any harmful one.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- I read the complete main text (Sections 1-8) and Appendices A through E, plus Appendix F.1; the weight-fitting, calibration and cohort-check details in Appendix F.2-F.5 were only partially inspected, so claims about how learned weights and score thresholds were fitted rest on the Section 7.4 summary and Appendix F.1.
- Figures 1-6 are known only through their captions and surrounding prose; I could not inspect the rendered plots, so every number attributed to Figures 5 and 6 comes from the Section 7.3-7.4 text.
- No citation could be verified offline; the paper's dated references (OpenAI 2026, ICML 2026, RewardBench 2 at ICLR 2026) are reported as the paper states them.
- The StrongREJECT construction models release as principal gain 1/2 minus median human harm with a cutoff fixed at 1/2 and never estimated; I record this as a convention of the paper rather than a measured quantity.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.