Single-Query Black-Box Calibration Auditing via Logit Bias
Roman Plaud, Antoine Saillenfest, Matthieu Labeau, Thomas Bonald, Willem Waegeman
cs.LG
Quick verdict
Clever one-query calibration audit of black-box LLMs via logit_bias thresholds; theory sound, simulated results strong, but no live-API test and no error bars.
Summary
The paper addresses calibration auditing of LLMs served through APIs that hide continuous output probabilities. Exploiting the logit_bias parameter still exposed by some providers, the authors add a threshold-dependent shift b = −ln(t/(1−t)) to the positive token's logit, so that under greedy decoding the emitted token exactly reveals the indicator 1(f(X) ≥ t) in a single query. Since one query per sample cannot verify both bounds of an ECE bin, the dataset is randomly split into M disjoint subsets whose adjacent pairs estimate each bin's contribution, yielding the ECEblind estimator. Under an L-Lipschitz calibration function, the authors prove MSE ≤ O(M³/N + 1/M²) against the True Calibration Error, with optimal bin scaling M ∝ N^(1/5). On BoolQ and a binarized MMLU, using four open-weight models (Qwen-2.5-7B, Llama-3.1-8B, Mistral-7B, Gemma-2-9B) to simulate opaque APIs, the method reaches average MAE near 0.01 versus a white-box oracle, matching iterative logit extraction (Carlini et al.) at K=5 queries for one-fifth of the cost, and clearly beating verbalized confidence and self-consistency sampling. Stated limitations: binary tasks only; reliance on a logit_bias feature that providers are deprecating; evaluation confined to simulated environments; the white-box oracle itself carries finite-sample and binning bias; small benchmarks cap the bin count (M=5 on BoolQ). The paper reports no code release, and the reported numbers appear to be single-run point estimates without confidence intervals.
Review complete
Assessment at a glance
4
risk flags identified
● high 0
● medium 3
● low 1
4 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- Any LLM API exposing logit_bias can be manipulated so that one greedy query per sample evaluates the exact threshold indicator 1(f(X) ≥ t) (Sections 1 and 4).
- ECEblind is a provably consistent estimator of the True Calibration Error, with MSE ≤ O(M³/N + 1/M²) when M ∝ N^α for 0 < α < 1/3 (Theorem 1, Appendix B).
- The optimal bin count for the single-query estimator scales as M ∝ N^(1/5) (Appendix C).
- At a one-query budget, ECEblind attains average MAE near 0.01 against the white-box oracle, matching iterative logit extraction at K=5 and clearly beating verbalized confidence and sampling (Section 5, Table 1).
- The Blind Calibration Curve provides signed, density-weighted diagnostics whose bar magnitudes sum exactly to ECEblind (Section 5, Appendix F).
Strengths
Exact, cleanly proven threshold mechanism
Section 4 and Appendix D.2 show exp(C) factors out of the biased softmax, so adding b=−ln(t/(1−t)) to the positive token set V+ makes the temperature-0 output an exact indicator 1(f(X)≥t); the multi-token-variant extension is proven in Appendix D.
Complete theoretical treatment
Appendix B derives a correct three-term bias decomposition (Bstat ≤ √2·M^(3/2)/N^(1/2), Btrap ≤ 1/2M, Bbin ≤ (1+L)/2M) plus an Efron–Stein variance bound of 8M²/N; Appendix C honestly discusses bound tightness and Appendix B the Lipschitz caveat.
Strong cost-accuracy trade-off at the one-query operating point
Unusually candid limitations section
Weaknesses
No empirical uncertainty quantification
Bin-count reporting inconsistency
Simulation-only external validity
Statistical inefficiency by design and narrow scope
K-cost claim slightly generous
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
This method paper targets a real gap: standard calibration metrics (ECE) require continuous token probabilities that commercial LLM APIs increasingly withhold. The authors propose ECEblind and claim that (1) any API exposing logit_bias can evaluate exact threshold indicators 1(f(X) ≥ t) in one greedy query per sample; (2) the resulting estimator is provably consistent for the True Calibration Error under an L-Lipschitz calibration function, with MSE ≤ O(M³/N + 1/M²) (Theorem 1); and (3) one query matches iterative logit extraction [1] at K=5. The scope is deliberately narrow — binary tasks with two target token sets — and the paper is transparent about this (Appendix A).
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 4 strengths & 5 weaknesses, analyzed in detail
- 4 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
How can you measure LLM calibration if the API hides probabilities?
The method adds a logit_bias shift b = −ln(t/(1−t)) to the 'True' token set and lets the API emit only two allowed outcomes (e.g., True/False) under greedy decoding. Whichever token comes back tells you exactly whether the model's hidden probability f(X) is at least t — one query per sample. Averaging such threshold tests over a binned, randomly partitioned dataset yields a provably consistent estimate of calibration error, called ECEblind.
Does this work on OpenAI, Anthropic, and Gemini today?
Only where logit_bias exists. Per the paper's limitations section, OpenAI supports logit_bias on standard models but disables it on newer reasoning models (the o1 series), Anthropic has removed the parameter from its API, and Google's Gemini API ignores it. Consequently, the paper's experiments simulate black-box APIs using open-weight models rather than querying live commercial endpoints.
How accurate is the single-query method compared to multi-query approaches?
Against a white-box oracle on BoolQ and binarized MMLU across four 7–9B open-weight models, ECEblind reaches an average MAE near 0.01 with one query per sample — roughly matching iterative logit extraction at K=5 queries, i.e., about one-fifth of the query cost. Verbalized confidence (≈0.074 average MAE) and 8-query self-consistency sampling are consistently less accurate.
Can the method handle multi-class tasks like MMLU's A/B/C/D options?
No. The derivation requires exactly two target token sets so the probability reduces to a sigmoid of a logit difference; applying one logit_bias value cannot isolate a threshold for one class among many. Multi-class calibration via single queries is stated as an open problem — the paper's MMLU experiments binarize each four-way question into independent Yes/No sub-questions.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Figures 1–4 appear only as image placeholders in the extraction; figure-level quantitative claims were cross-checked against Table 1 and the running text, but plot details could not be visually inspected.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.