BLOOM-WILT: Logit Tilting for Behaviour Elicitation in Automated LLM Auditing
Adrians Skapars, Edoardo Manino
cs.AI · cs.CL
Quick verdict
BLOOM-WILT elicits rare LLM behaviours via input refinement plus logit tilting, beating BLOOM in 30/32 settings; strong but small-scale and selection-prone.
Summary
BLOOM-WILT extends the BLOOM automated auditing pipeline with optimisation pressure at both ends of the auditor–target interaction. On the input side, G-PAIR rewrites the auditor's opening message and strategy across sampling rounds using prior judge scores; on the output side, LogitTilt samples tokens from softmax(ℓ_tgt + β·ℓ_beh), mixing the target's own next-token distribution with its distribution under a behaviour-eliciting prompt, plus a naturalness floor (τf=1e-4). On Qwen3.5-4B self-harm encouragement (100 scenarios, 3 turns), behaviour presence rises from 51.0% (best-of-N) to 100.0% with WILT and 99.5% with LogitTilt alone, while arithmetic output probability stays at or above vanilla (56.0% vs 51.1%) and geometric mean improves (40.0% vs 27.5%); output-side methods beat input-side ones (BEAST-in 31.7%, FLRT 33.5%, G-PAIR 66.2%). Across 4 open-weight ~3–4B targets × 8 behaviours, WILT beats vanilla BLOOM in 30/32 settings and ranks highest in 24, overturning BLOOM's model-safety rankings; swapping in a Claude-Sonnet-4.6 judge preserves the ordering. Limitations: requires logit access; fails where the elicitation prompt triggers refusal (Gemma-4-E4B self-harm, 32.2% vs vanilla 34.6%); targets are small models; results depend on LLM judges and post-run transcript selection on the evaluation set.
Review complete
Assessment at a glance
5
risk flags identified
● high 0
● medium 2
● low 3
5 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- LogitTilt elicits behaviours by sampling from softmax(ℓ_tgt + β·ℓ_beh) — the target's own logits mixed with its logits under a behaviour-eliciting prompt — requiring no training, no second model, and access only to next-token distributions.
- Output-side steering beats input-side prompt optimisation at matched compute: LogitTilt reaches 99.5% behaviour presence vs 31.7% (BEAST-in), 33.5% (FLRT) and 66.2% (G-PAIR) on Qwen3.5-4B self-harm encouragement (Table 1).
- WILT beats vanilla BLOOM in 30 of 32 (model, behaviour) settings across 4 open-weight 3–4B targets and 8 behaviours, attaining the highest presence in 24 and overturning BLOOM's model-safety rankings (Figure 4, Table 8).
- A single hyperparameter β, plus free post-run selection over already-sampled rounds, traces the elicitation–plausibility Pareto frontier (Figure 3).
- Elicited responses are model-specific and transfer poorly across targets (Table 9), indicating the auditor adapts to the target rather than reciting generic attacks.
Strengths
Plausibility is never traded away, and this is measured, not asserted
Table 1: LogitTilt/WILT hold arithmetic output probability above vanilla best-of-N (54.4/56.0 vs 51.1) and raise geometric mean (37.1/40.0 vs 27.5); the naturalness floor holds minimum token-prob at 1.3e-2 vs 1.8e-5 for BoN (Table 5), catching degenerate loops.
Unusually thorough component and construction ablations
Table 3 isolates elicited-only (±floor), abliterated elicitor (presence drops to 58.8%), small-sibling elicitor (68.1%), both-small (87.8%), and the W2S difference construction (worst, 24.3% Pareto); Table 5 ablates compliance prefill and floor individually.
Judge and generator robustness actually tested
Specificity, transfer, and benign controls
Anti-cherry-picking and reproducibility detail
Weaknesses
Headline setting is saturated
Post-run selection operates on the evaluation set
Compute matching is wall-clock, not token-level
Only 3–4B targets; scale generalisation untested
LLM-judged metrics with LLM-generated rubrics
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
BLOOM-WILT targets automated behavioural auditing: given a one-sentence behaviour description, find natural multi-turn transcripts in which a target model spontaneously exhibits the behaviour. The paper claims three things: (1) output-side decoding interventions beat input-side prompt optimisation for behavioural elicitation at matched compute (Table 1); (2) WILT — G-PAIR input refinement plus LogitTilt output steering — beats vanilla BLOOM in 30 of 32 (model, behaviour) settings across 4 open-weight ~3–4B targets and 8 behaviours (7 harmful, 1 benign control), overturning BLOOM's model-safety ranking (Section 5.7, Figure 4, Table 8); (3) a single steering strength β traces the elicitation–plausibility Pareto frontier without pushing output probability below baseline. The headline number — 51.0% → 100.0% behaviour presence for self-harm encouragement on Qwen3.5-4B — is stated consistently in abstract, contributions and conclusion.
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 5 strengths & 5 weaknesses, analyzed in detail
- 5 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What is BLOOM-WILT and how is it different from ordinary jailbreaking?
BLOOM-WILT is an automated auditing pipeline that starts from a one-sentence behaviour description and searches for realistic multi-turn conversations in which a language model spontaneously exhibits that behaviour. Unlike classic jailbreaks that optimise a prompt against a refusal, it keeps both the simulated user's inputs (via G-PAIR iteration across rounds) and the model's own outputs (via LogitTilt decoding) plausible, so the resulting transcripts resemble interactions a real user could have.
Does LogitTilt need access to the model's weights?
No training or weight access is needed, but it does require the model's next-token logits, so it cannot be applied to text-only APIs. The steering mixes the model's own next-token distribution with its distribution under a behaviour-eliciting system prompt (z = ℓ_tgt + β·ℓ_beh), plus a naturalness floor that masks very improbable tokens; both distributions come from the same model.
What were the paper's headline results?
On Qwen3.5-4B self-harm encouragement, WILT raised judged behaviour presence from 51% (vanilla best-of-N BLOOM) to 100% at matched compute, with output probability at or above the baseline's. Across 4 open-weight 3–4B models and 8 behaviours, WILT beat vanilla BLOOM in 30 of 32 settings and produced a different model-safety ranking than vanilla BLOOM.
Is the code and data available, and what hardware is needed?
The paper links a GitHub repository (github.com/AdrSkapars/bloom-wilt), builds on the open-source BLOOM pipeline, and uses open-weight models for targets and the local auditor, with seeds and hyperparameter grids documented in the appendices. Reproduction requires roughly two 48GB GPUs; a single best-of-N run of 100 scenarios takes about 50 minutes on that setup.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Inline figure graphics (Figures 3, 5, 7, 8) are garbled by PDF text extraction — axis labels and legend text are jumbled — but all numeric tables (1, 2, 3, 5, 7, 8, 9, 10) are legible and were used for this review.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.