arXiv paper2026-09-10· machine learning (NLP) — uncertainty quantification for large language models· method paper

ActMap: Single-Pass Uncertainty Quantification from Generation-Time Activation Maps

cs.AI

Quick verdict

ActMap compresses generation-time hidden states into a fixed 96 KiB map; strong in-domain results, but near-chance transfer limits deployment claims.

Summary

ActMap is a single-pass uncertainty-quantification representation for LLMs. During one greedy generation, forward hooks record every layer's hidden state at every generated token; twelve temporal statistics, adaptive pooling (layers to 32, hidden dim to 128), and per-channel standardization compress the variable L×T×D trajectory into a fixed 12×32×128, 96 KiB map whose geometry is identical across generators. A 2.4M-parameter Vision Transformer maps it to p(correct). On TriviaQA, NQ-Open, direct-answer GSM8K, and CNN/DailyMail with Qwen3-8B, Llama-3.1-8B, and Mistral-7B under balanced splits, ActMap attains mean AUROC .825 across twelve pairs, beating all sampling, token-probability, attention, and embedding baselines, and matching ACT-ViT (.823), which trains on a 67x larger dense tensor, with lower ECE on ten of twelve pairs. Ablations attribute the signal to cross-layer, pooled-coordinate structure: last-token channels alone reach .879 versus .887 for the full map, a capacity-matched MLP matches the ViT, and coordinate permutation is the most damaging control. Key limitations, honestly reported: transfer across tasks, generators, and model scales is near chance (.510-.520 AUROC), so every deployment regime needs fresh labels; GSM8K calibration is weak (ECE up to .146); and under prevalence shift ECE reaches .15-.24. Code and the 476,372-map dataset are promised upon publication.

Review complete

Assessment at a glance

machine learning (NLP) — uncertainty quantification for large language models

risk flags identified

● high 0

● medium 2

● low 3

Reproducibility readinessmedium

Code & data availablenot stated

Review confidencehigh

AI text signal0/20 passages flagged

Deep research viableCode-reproducibleNeeds GPU

5 strengths · 5 weaknesses · 5 author questions

AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.

Levels reflect verification complexity — not a quality score. Evolve never rates your paper.

Key claims under review

A fixed 12x32x128 map of twelve temporal-statistic channels, captured during a single generation pass via forward hooks with no measurable overhead, preserves the uncertainty signal of the full L×T×D hidden-state trajectory.
Trained in-domain, a 2.4M-parameter ViT on these maps attains mean AUROC .825 across twelve 7-8B (model, dataset) pairs, leading all eight baselines and matching ACT-ViT (.823) from a 67x smaller artifact with lower ECE on ten of twelve pairs.
The predictive signal lies in cross-layer, pooled-coordinate structure rather than in temporal statistics — which mainly fix the map's shape — and not in the classifier architecture, since matched MLPs perform comparably.
Frozen-detector transfer across tasks, generators, and model scales is near chance (.510-.520 AUROC), so each deployment regime requires its own labeled generations.
Under class-prevalence shift toward natural accuracy, ECE degrades to .15-.24 and temperature scaling does not repair it, requiring prior correction.

Strengths

Fixed-shape, generator-agnostic artifact with measured negligible overhead
The 12x32x128 float16 map (96 KiB) is invariant across L in {32,36,64} and hidden sizes; hooked vs unhooked decoding both averaged ~7 ms/prompt (Cost section) and scoring is .024 ms/map, making the single-pass claim concrete rather than asserted.
Unusually rigorous strongest-baseline comparison
ACT-ViT is reproduced with its complete published 24-configuration architecture sweep on identical balanced splits and seeds (Appendix C.1), rather than as a cherry-picked fixed-score baseline; result is near-parity (.825 vs .823 mean AUROC) from a 67x smaller representation.
Ablations genuinely isolate the signal source
Honest negative results and failure analysis
Reproducibility protocol documented to an uncommon depth

Weaknesses

Transfer failure undercuts the deployment framing
Temporal channels add almost no signal despite the headline narrative
Occlusion localization rests on tiny effect sizes
Noisy programmatic labels, never human-audited
GSM8K is the weak regime and the setting is unrealistic for math

Full Review Report

$1.99 · one-time

Unlock the complete preliminary analysis Evolve produced for this paper.

Report preview

Scope & Claims

ActMap targets answer-level uncertainty quantification (UQ) for decoder-only LLMs: from one greedy generation, produce p(correct) using internal activations alone. Forward hooks capture the hidden state of every layer at every generated token; contiguous adaptive pooling reduces hidden dims to 128, twelve temporal statistics summarize the token axis, layers are pooled to 32 rows, and per-channel standardization yields a fixed 12x32x128 float16 map (96 KiB) shared in geometry across generators (Method; Appendix A). A 2.4M-parameter ViT over 4x16 patches is the default classifier. Central claims: (1) in-domain, ActMap leads all eight baselines on all twelve 7-8B (model, dataset) pairs; (2) it matches ACT-ViT at mean AUROC .825 vs .823 from a 67x smaller artifact, with lower ECE on ten of twelve pairs; (3) the signal is cross-layer, pooled-coordinate structure; (4) transfer across tasks, generators, and scales is near chance. Claims are mostly well-scoped, with the abstract's 'practical primitive for scalable oversight' the most expansive phrasing.

Methodology Assessment

The evaluation protocol is unusually disciplined. Baselines span an explicit black-/grey-/white-box taxonomy (Table 2); ACT-ViT is reproduced with its full 24-configuration sweep on identical balanced splits and seeds (Appendix C.1) — more diligence than most comparison papers. Supervised methods share train/validation/test rows. Labeling is programmatic: alias exact-match or token-F1>=0.8 for QA, numeric match for GSM8K, MiniCheck (Flan-T5-Large) at .5 per sentence for CNN/DailyMail — cheap but noisy and never human-audited. The main internal tension: Table 5 shows l

The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.

All 5 strengths & 5 weaknesses, analyzed in detail
5 risk flags with severity, section & reviewer questions
Methods & statistical evidence assessment
5 questions a peer reviewer would ask
Full six-dimension evaluation report & recommendations

One-time purchase · this paper only

One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.

Frequently asked questions

What is ActMap and how does it detect unreliable LLM answers?

ActMap records the hidden states of every transformer layer at every generated token during a single generation pass, compresses them into a fixed 12x32x128 tensor of temporal statistics (96 KiB), and a small 2.4M-parameter Vision Transformer classifier reads an estimated correctness probability from that tensor. It needs no extra model calls and no sampled regenerations.

How accurate is ActMap compared to other uncertainty methods?

On four tasks (TriviaQA, NQ-Open, direct-answer GSM8K, CNN/DailyMail summarization) with three 7-8B open models, ActMap reaches mean AUROC .825 across twelve model-dataset pairs — above sampling-based semantic entropy, token-probability, attention-based, and embedding baselines — and roughly equals ACT-ViT (.823), which trains on a 67x larger dense activation tensor, with lower calibration error on ten of twelve pairs.

Can a detector trained on one model or task be reused on another?

Mostly no. The paper's frozen-detector transfer tests show AUROC near chance (.510-.520) when moving across generators, tasks, or model scales (8B to 32B); only TriviaQA-to-NQ-Open transfer within QA stays strong (.837). Each deployment regime requires its own labeled generations.

Is the ActMap code or dataset available?

The authors state that code, replication materials, and the 476,372-map dataset will be released on Hugging Face upon publication. At the time of this review nothing has been released yet, though the appendix documents environment, hyperparameters, and determinism settings in unusual detail.

About this review

This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.

Figures 1-4 appear only as captions in the extracted text; the occlusion atlas, risk-coverage curves, and cost-quality plot could not be visually inspected, so figure-based judgments rely on the surrounding prose and tables.

Go deeper on this paper

Wrote a paper? Stress-test it before reviewers do.

Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.

Scan your paper