Bridging the Confidence Gap: Temperature Scaling for Calibrating Test-Time Prompt Tuning
Yuwei Liang, Jian Liang, Dapeng Hu, Yinuo Xu, Ran He
cs.LG
Quick verdict
CoTS/E-CoTS cut TPT's calibration error via zero-shot-anchored temperature scaling, keeping accuracy; evidence broad, though ECE mostly mirrors the anchor.
Summary
This method paper targets miscalibration in test-time prompt tuning (TPT) for CLIP-style vision-language models. A pilot study (Confidence Replacement, CoR, Sec. 2.2) first shows that TPT's miscalibration stems mainly from confidence inflation: substituting zero-shot confidence for TPT confidence sharply reduces ECE. The authors then propose CoTS, a post-hoc temperature scaling that optimizes a per-sample scalar temperature over selected strongly augmented views to align TPT's top-1 confidence with the well-calibrated zero-shot confidence, leaving argmax predictions (and hence accuracy) unchanged. An optional weak-strong ensemble whose weight is set adaptively by the average cosine similarity of text embeddings yields E-CoTS, boosting accuracy while CoTS re-calibrates. On eleven fine-grained datasets and four ImageNet variants with ViT-B/16 (plus ResNet-50 and SigLIP), CoTS cuts TPT's average ECE from 11.30% to 4.28% and from 11.90% to 5.30% respectively with no accuracy change; E-CoTS raises ImageNet-variant accuracy from 60.74% to 62.95% at 5.38% ECE. Plug-and-play tests with C-TPT, Penalty, O-TPT, A-TPT, SoC, TTL, TPS and online TDA, three-seed standard deviations, and extra metrics (Brier score, class-wise and adaptive ECE) support generality. Main limitations: CoTS largely inherits the zero-shot anchor's residual miscalibration (e.g., 8.34% zero-shot ECE on ImageNet-A), fine-grained ensemble gains are marginal (+0.03%), and no significance testing is reported.
Review complete
Assessment at a glance
5
risk flags identified
● high 0
● medium 2
● low 3
5 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- TPT's miscalibration stems mainly from confidence inflation: replacing TPT confidence with zero-shot confidence (CoR pilot, Sec. 2.2) substantially reduces ECE, beating SaLS and C-TPT.
- CoTS, a post-hoc temperature optimized per sample to match zero-shot confidence over top-6 confident strong views, reduces TPT's average ECE to near the zero-shot level (11.30%→4.28% fine-grained; 11.90%→5.30% ImageNet variants) without changing any prediction or accuracy.
- A weak-strong ensemble weighted by average text-embedding cosine similarity boosts accuracy (ImageNet variants 60.74%→62.95%), and re-applying CoTS (E-CoTS) keeps calibration competitive (5.38% ECE).
- CoTS/E-CoTS are plug-and-play: combined with C-TPT, Penalty, O-TPT, A-TPT, SoC, TTL, TPS or online TDA, and across ViT-B/16, ResNet-50, SigLIP and CoOp initialization, they generally improve accuracy, ECE, or both (Tables 3-13).
Strengths
Sharp motivational analysis isolating the root cause
The Confidence Replacement pilot (Sec. 2.2, Fig. 2) cleanly demonstrates that TPT's miscalibration is confidence inflation, not decision change: swapping in zero-shot confidence already beats SaLS and C-TPT on ECE, which then directly motivates the temperature-alignment design.
Accuracy preservation verified, not just asserted, across many settings
Tables 1, 2, 6, 8, 9, 12, 13 show CoTS accuracy is bit-identical to TPT on ViT-B/16, ResNet-50 and SigLIP across 15 datasets; ECE drops are large (fine-grained 11.30%→4.28%; ImageNet variants 11.90%→5.30%) while ResNet-50 ECE falls 17.55%→6.87%.
Unusually broad generality tests for a post-hoc fix
Thorough ablations and supplementary metrics
Honest cost reporting and seed variance
Weaknesses
CoTS inherits the zero-shot anchor's calibration ceiling
Accuracy preservation is by construction, not a finding
Fine-grained ensemble benefit is marginal
Statistical reporting is informal
Listed baselines never appear in results
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
This method paper addresses miscalibration in episodic test-time prompt tuning (TPT) of CLIP-style vision-language models. The central diagnosis (Sec. 2) is that TPT boosts accuracy over zero-shot CLIP (e.g., 60.74% vs 57.22% on ImageNet variants, Table 2) but inflates confidence, raising ECE from 4.98% to 11.90%. A pilot study — Confidence Replacement (CoR, Sec. 2.2) — replaces TPT confidence with zero-shot confidence per sample and already outperforms SaLS and C-TPT on ECE, establishing that miscalibration is a confidence-shift problem, not a decision problem. The proposed CoTS (Sec. 3.1) turns this into a consistent post-hoc fix: a per-sample scalar temperature tau is optimized for 50 steps to match temperature-scaled TPT top-1 confidence to zero-shot top-1 confidence over the top-6 confident strongly augmented views. E-CoTS (Sec. 3.2) adds a weak-strong ensemble whose weight alpha equals the mean pairwise cosine similarity of the text embeddings. The headline claims: near-zero-shot calibration without accuracy loss, plus accuracy gains from the ensemble.
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 5 strengths & 5 weaknesses, analyzed in detail
- 5 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What is CoTS in simple terms?
CoTS is a post-hoc fix for calibrating test-time prompt tuning (TPT) of CLIP-like models. After TPT adapts the model to a single test image, CoTS tunes one temperature parameter for that image so the model's top confidence matches what the original zero-shot CLIP would have predicted confidently. Because temperature scaling never changes which class is predicted, accuracy stays exactly the same while the confidence numbers become better calibrated (lower expected calibration error).
Does CoTS improve accuracy?
No — by design, CoTS leaves every prediction unchanged, so accuracy is identical to TPT. The companion method E-CoTS adds a weak-strong augmentation ensemble (weighted by text-embedding similarity) on top of CoTS, which raises accuracy, for example from 60.74% to 62.95% on ImageNet variants, while keeping calibration error low (5.38% ECE).
What datasets and models were used to evaluate CoTS?
Eleven image classification datasets (ImageNet, DTD, Flowers102, Food101, SUN397, Aircraft, OxfordPets, Caltech101, UCF101, EuroSAT, StanfordCars) plus four shifted ImageNet variants (ImageNet-A, -V2, -R, -Sketch). Backbones are CLIP ViT-B/16 and ResNet-50, plus SigLIP, with experiments under CoOp prompt initialization and an online TTA framework (TDA).
Is the CoTS code available?
The paper states that code is available at github.com/yuweiliang911/CoTS. This review did not independently verify that the repository exists or runs; all hyperparameters (64 AugMix views, top-10% selection, one AdamW step at lr 0.005, 50 temperature steps, K=6, T0=1.0) are specified in the paper, which supports reimplementation.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Figures (1-9) appear in the extraction only as axis tick text; trend plots (Figs. 2-8) and reliability diagrams (Fig. 9) could not be visually inspected, so CoR comparisons in Fig. 2 are taken from surrounding text only.
- Table 7 (computation time) is garbled in extraction; the exact per-method runtime attribution is uncertain beyond the stated ~+0.05 s/sample overhead for CoTS.
- The appendix may be truncated: ZS-Norm (listed in Sec. 4.1) and D-TPT (Sec. B) never appear in any table; their absence may be an extraction artifact rather than an omission by the authors.
- This is an arXiv v1 preprint (2609.17386, Sep 2026); no peer-review verdict from a venue is reflected in this review.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.