Inherited Heads: Audio language models track speakers with their text backbone's attention, and an attention-mass ranking retrieves a different set
cs.LG · cs.CL · cs.SD
Quick verdict
Claims audio-LM speaker tracking runs on text-inherited heads and attention-mass ranking retrieves a worse set; rigorous and candid about its limits.
Summary
Das asks whether audio language models (ALMs) possess speaker-tracking machinery of their own. On 500 six-speaker LibriSpeech strips, three ALMs (Qwen2-Audio, Ultravox, SALMONN) name the queried speaker on only 6–16% of trials; yet biasing the attention logits of 100 heads — under a tenth of each model, with no training — redirects the description to a chosen speaker on 90.7–99.0% of trials. Those heads are largely inherited: ranking heads on each model's text-only backbone (or a same-family model for Qwen2-Audio) on a written version of the task yields sets sharing 66–74 of 100 heads with the audio sets (layer-matched chance about 20), and the text-chosen heads steer the audio models at 80.8–95.0%. The paper is careful that an equal-size random draw from the same discovered hundred does nearly as well, so sharedness-as-cause is explicitly not established. A second finding contrasts the established attention-mass ranking with a per-head-normalized selectivity variant: their top-100s share 69, 37 and 4 heads; on Ultravox, mass-ranked heads leave 69.7% of output unattributable (1.0% for selectivity), though a depth confound is uncontrolled and this is one arm of six. Instrument audits show the edit at saturation mostly reallocates attention from prompt and sink, and random-head controls vary widely across draws. Limitations: 4-bit quantization, no audio positional control, and transfer numbers measured on the discovery split.
Review complete
Assessment at a glance
5
risk flags identified
● high 0
● medium 3
● low 2
5 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- Biasing the attention logits of 100 heads (under a tenth of each model, with no training) redirects ALM speaker descriptions to a chosen target on 90.7–99.0% of trials across Qwen2-Audio, Ultravox and SALMONN, against a 6–16% baseline.
- Top-100 heads ranked on the text-only backbone — or a same-family text model for Qwen2-Audio — steer the audio models at 80.8–95.0%, and audio/text head sets share 66–74 of 100 indices against a layer-matched chance near 20.
- The shared subset is causally sufficient for steering, but sharedness itself is not established as the cause: an equal-sized random draw from the same discovered hundred performs comparably on all three models.
- Ranking by attention mass versus per-head-normalized selectivity retrieves partly different sets (69/37/4 of top-100 shared); on Ultravox the mass-ranked heads leave 69.7% of output unattributable versus 1.0% for selectivity, with depth as an uncontrolled confound and this being one arm of six.
- Methodologically: attention-logit steering at saturation mostly reallocates attention from prompt tokens and the sink rather than other segments, and matched random-head controls are distributions wide enough to flip single-draw comparisons.
Strengths
Control methodology beyond genre standard
Layer-matched vs band-matched random draws, exclusion of the discovered hundred, and 3–10 draws printed in full (Appendix A, Tables 4–7). Section 6 turns control variance into a finding: five identically constructed draws span 0.008–0.650 accuracy on Ultravox at 29 heads, and ten Ultravox draws at K=100 span 0.007–0.663 (Table 7), showing single-draw comparisons can carry the wrong sign.
Cleanest cross-modality transfer test
Section 4 ranks heads on the text-only backbone (or Qwen1.5-7B-Chat, a same-family model whose weights differ twice over) and steers audio models at 95.0/85.0/80.8% against band-matched randoms of 12.5/45.0/15.0%; overlaps of 66–74/100 sit far above layer-matched nulls (19.5–24.5, 95th pct 25–30), and text-derived heads land within 0.008 of the audio-chosen set on Qwen2-Audio.
Instrument honesty that constrains its own mechanism claims
Disciplined non-reproduction (Section 8)
Number traceability
Weaknesses
4-bit quantization undermines every head ranking
Ultravox failure arm lacks its needed control
No positional control for the audio arms
Partial in-sample evaluation
Testbed idiosyncrasies and no code release
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
The paper asks whether audio language models (ALMs) track speakers with machinery of their own. Three instruction-tuned models — Qwen2-Audio-7B-Instruct, Ultravox-v0.6-8B (frozen Llama-3.1-8B backbone) and SALMONN-13B (LoRA-adapted Vicuna-13B) — are tested on 33.5-second strips of six 5-second LibriSpeech segments by six speakers, prompted to describe what one named speaker says; baseline accuracy is 6–16% against 16.7% chance (Section 1). Four claims: (1) a fixed +B/−B bias on the pre-softmax attention logits of 100 heads, under a tenth of each model and with no training, redirects the description at 90.7–99.0%; (2) heads ranked on the text-only backbone (or a same-family model for Qwen2-Audio) transfer at 80.8–95.0%, with audio/text top-100s sharing 66–74 heads against a layer-matched null near 20 (Figure 2); (3) the shared subset is causally sufficient (Table 1), but the paper explicitly refuses to claim sharedness is causal, because an equal-size draw from the same discovered hundred lands within 0.039 on two models and above the shared part on SALMONN; (4) mass and selectivity rankings share only 69/37/4 of their top-100, with a severe Ultravox failure arm (69.7% unattributable output vs 1.0%). Two self-audits round this out: at saturation (B=5) the edit pulls most gained attention from prompt (0.416) and sink (0.412) rather than the other segments (0.173) (Section 6, Figure 5); and layer-matched random controls are distributions, not points — five draws span 0.642 of accuracy at 29 heads on Ultravox.
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 5 strengths & 5 weaknesses, analyzed in detail
- 5 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What is the paper 'Inherited Heads' about?
It shows that audio language models track which speaker to describe using attention heads largely inherited from their text-only backbone. Steering 100 such heads with a fixed attention-logit bias makes the model describe a chosen speaker 90.7–99.0% of the time (baseline 6–16% versus 16.7% chance), and heads chosen without ever seeing audio still work, at 80.8–95.0%.
Does the paper prove the text/audio shared heads are special?
No. The shared subset is sufficient for steering, but an equal-sized random draw from the same discovered set of 100 heads does nearly as well on all three models, so the authors explicitly do not claim that being shared with the text backbone is what makes the heads effective.
Why does the attention-mass ranking fail on Ultravox?
On one of six arms, the mass-ranked top-100 heads leave 69.7% of output unattributable (versus 1.0% for a per-head-normalized selectivity ranking). The paper cautions that the two head sets sit about 17 layers apart and no depth-matched control was run, so the failure cannot yet be attributed to the score itself rather than to head depth.
Is code or data available for this paper?
The data and models are public (LibriSpeech-derived six-speaker strips, a released comic-strip corpus, and three open-weight audio models), and figures are generated from a run-output ledger with a build guard, but the paper does not state a code release.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Figures are text-extracted and partially garbled; all cited numbers were cross-checked against tables and body prose rather than figure pixels.
- The reference list is dense with 2026 preprints and workshop papers that could not be verified.
- Full text appears complete (abstract through Appendix A), so truncation is unlikely, but figure-internal details rest on the prose descriptions.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.