In-Context Neurofeedback: Can LLMs Control Their Internal Representations through Privileged Access?
Koshiro Aoki, Ryota Takatsuki, Gouki Minegishi, Yusuke Haruki, Daisuke Kawahara
cs.AI
Quick verdict
Proposes privileged-access neurofeedback test of LLM self-control; effects small, inconsistent across 4 models—prior claims unproven, but this null isn't conclusive.
Summary
This paper asks whether large language models can deliberately alter their own internal representations when the control target cannot be inferred from text. Revisiting Ji-An et al.'s (2025) neurofeedback result, the authors argue its control targets are inferable from prompts, so apparent control could come from ordinary text-pattern extension rather than metacognition. They propose in-context neurofeedback (ICN): the model must repeat one fixed sentence while trying to maximize a 0-100 score computed by a logistic-regression probe on its hidden activations, with the scoring rule withheld. Because the output text is constant and the rule hidden, the control target satisfies the 'privileged access' requirement analogous to human decoded neurofeedback. Across four open-weight models (Llama-3.1-8B/70B-Instruct, Qwen3-8B/32B), three datasets (SST, ETHICS commonsense, True-False), five layers, and two metrics, only 45/120 settings showed significant label-1-versus-label-0 differences after Benjamini-Hochberg correction, concentrated in SST (24/40) and Qwen3-8B (19/30), with small effect sizes (Cohen's d, h < 0.5). An ablation revealing the scoring rule barely changed results (38/120). The authors conclude that current LLMs do not demonstrate reliable privileged control and that prior positive claims cannot exclude superficial mechanisms; since task success is only a sufficient condition, failure here cannot rule out the capability. Code is released.
Review complete
Assessment at a glance
3
risk flags identified
● high 0
● medium 2
● low 1
5 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- Control targets in Ji-An et al.'s (2025) explicit and implicit (prefilled) neurofeedback tasks are inferable from text, so their reported LLM control cannot exclude a trivial text-pattern mechanism.
- ICN fixes the output sentence and withholds the probe-based scoring rule, making the control target privileged and task success a sufficient condition for metacognitive control.
- Across 120 settings (4 models x 3 datasets x 5 layers x 2 metrics), only 45 show significant label-1 vs label-0 differences after BH correction, concentrated in SST (24/40) and Qwen3-8B (19/30), with all effect sizes below 0.5.
- Revealing that the score derives from an activation probe does not restore control (38/120 significant), so rule hiddenness is not the reason models fail.
- Rigorous assessments of LLM metacognition should demand privileged access; current open-weight LLMs do not demonstrate reliable privileged control.
Strengths
Clear conceptual contribution
Section 2 operationalizes 'strict privileged access' (following Song et al. 2025b) for the control setting; Section 3.2 and Appendix B show that both the explicit imitation task and the prefilled implicit task of Ji-An et al. (2025) fail this criterion, so their probe shifts cannot distinguish metacognitive from trivial text-pattern mechanisms.
Above-norm statistical rigor
Prespecified one-sided paired t-tests and exact McNemar tests paired by sentence, Benjamini-Hochberg correction across all 120 tests at alpha=0.05, and Cohen's d/h with 95% CIs; Appendix F Table 3 reports every p and adjusted q rather than only winners.
Useful controls and ablations
Honest scoping of the negative result
Reproducibility hygiene
Weaknesses
Primary hypothesis excludes its own control arm
Feedback-channel quality confound unanalyzed
Task may be near-impossible by construction
Internal vs behavioral evaluation asymmetry
Abstract framing slightly ahead of the evidence
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
The paper re-examines whether LLMs can *control* — not merely report on — their internal representations. Its central conceptual move is importing the 'privileged access' criterion (Section 2, following Song et al. 2025b) into the control setting: a target counts as privileged only if an external observer seeing inputs and outputs cannot recover it. The authors apply this lens to Ji-An et al. (2025), showing that both the explicit imitation task and the prefilled implicit task (Appendix B) leave the target inferable from text, so observed probe shifts could reflect ordinary text-pattern extension (the 'trivial mechanism') rather than metacognition. Their ICN paradigm (Section 3.3) fixes the output sentence, hides the scoring rule, and feeds back only an integer 0-100 from a logistic-regression probe on mean-pooled hidden activations, making success a sufficient condition for privileged control (Section 6.1). Claims are appropriately hedged: the headline is that control is *not demonstrated* under the stricter setting, not that the capability is absent.
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 5 strengths & 5 weaknesses, analyzed in detail
- 3 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
Did the LLMs pass the in-context neurofeedback test?
Mixed, leaning negative. In 45 of 120 tested settings the label-1-rewarding condition ended significantly higher than the label-0-rewarding condition after multiple-comparison correction, but results were inconsistent across models and datasets (concentrated in SST and Qwen3-8B) and all effect sizes were small (below 0.5), which the authors read as no reliable demonstration of privileged control.
What does 'privileged access' mean in this paper?
Knowledge of a state that an external observer cannot recover from the model's inputs and outputs alone. Here, because the output sentence is fixed and the scoring rule is hidden, neither the fixed sentence nor the score history reveals which internal feature is being scored, so successfully changing the score would require access to internal representations.
What problem did the paper find with the earlier Ji-An et al. (2025) result?
In their tasks the model can raise the probe score simply by producing text whose surface features match the target label (e.g., obviously prosocial sentences), because the control target is inferable from the prompt. Even their prefilled 'implicit' task does not fix this, since the model can internally plan label-consistent content before the output is overwritten.
Is code and data available for this paper?
Yes. The authors state the code is at github.com/AokiKoshiro/in-context-neurofeedback, and all experiments use public datasets (SST, ETHICS commonsense, Azaria-Mitchell True-False) and open-weight models (Llama-3.1-8B/70B-Instruct, Qwen3-8B/32B).
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Quantitative trajectories exist only inside figures; endpoint statistics (Table 3), effect-size bounds, and aggregate rates (73.1% mean probe accuracy, 93.7% compliance) are the machine-readable evidence in the text, so trajectory-level judgments here rest on figure captions and authors' descriptions.
- Some figure axis annotations are garbled by PDF extraction (e.g., Figure 9b, Llama-3.1-70B layer-0 SST probe output); no number cited in this review derives from those garbled axes.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.