The Dice Roll Method: A Standardized Protocol for Repeated-Query Auditing of Large Language Model Brand Recommendations
cs.IR · cs.CL
Quick verdict
A statistically principled protocol for repeated-query LLM audits with credible external validation, but all data is the author's own and commercially tied.
Summary
The paper formalizes the 'Dice Roll Method' — querying an LLM repeatedly with identical prompts and aggregating brand recommendations — as a standardized auditing protocol. Grounded in a generative model of temperature-scaled nucleus sampling, it decomposes response variance into token-level, prompt-phrasing, model, and version/time components, replacing the preliminary version's normal-theory stack with negative-binomial GLMMs, Cliff's delta with BCa intervals, moving-block and parametric bootstraps, simr-based power analysis, and generalizability-theory G/D studies. Reanalysing five prior studies by the same author (~190,000 observations claimed; 270+ brands, 6 languages, 5-40 iterations), it derives iteration tiers (n=5, G=0.58 exploratory; n=10, G=0.74 confirmatory; n=15, G=0.81 rigorous), finds count-, set-, embedding-, and fairness-based (PASOR) metric families complementary, and shows diminishing-returns convergence (80% of the n=10 precision ceiling at n=7). A pre-registered external validation on three independent corpora (Motoki 100-round; Rozado 24 models; llm-stability) replicates D-study predictions in 37 of 39 cells, but fixed iteration tiers do not transfer, motivating a 'pilot-then-solve' prescription; two drift-battery components fail and are amended. Main limitations: every core dataset is the author's own, the author is CEO of a company commercializing the protocol, brand-domain external validation remains open, and the pooled power simulation is admitted to be mis-specified at the per-cell level.
Review complete
Assessment at a glance
5
risk flags identified
● high 1
● medium 2
● low 2
4 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- Repeated LLM queries are dependent, over-dispersed, and potentially non-stationary, so the valid inferential stack is NB-GLMM plus Cliff's delta plus dependence-preserving bootstrap plus G-theory, not i.i.d. normal-theory tests.
- Three iteration tiers (n=5/10/15 giving G=0.58/0.74/0.81) are defensible on the authors' data, but external validation shows fixed tiers do not transfer, so the real prescription is pilot-then-solve: estimate variance components on a 10-iteration pilot and solve G(n) >= 0.80.
- Count-, set-, embedding-, and fairness-based (PASOR) metric families are complementary, collapsing to a recommended three-metric core (Gini or CV, ensemble cosine similarity, PASOR-Gini) because structural count metrics are nearly redundant on narrow prompt universes.
- The pre-registered external validation replicates D-study reliability predictions in 37 of 39 cells and the n=5 power value to two decimals, and confirms non-determinism at temperature 0 in every llm-stability cell, while falsifying fixed iteration tiers and two drift-battery components.
Strengths
Self-critical revision with explicit defect accounting
Section 1 itemizes the preliminary version's failures (i.i.d. t-tests, Cohen's d on over-dispersed counts, ordinary bootstrap, asserted log-convergence), and Section 2 derives why autoregressive dependence, over-dispersion, and non-stationarity invalidate each; every replacement tool (NB-GLMM Eq. 4, Cliff's delta, block bootstrap, G-theory) is tied to a named variance component in Eq. 3.
Pre-registered external validation reporting failures at full prominence
Table 16 lists 39 verdicts: EV1b (single-facet G(5) < 0.80) fails in all three corpora, EV3a/b show convergence families are horizon-relative (on C1's 100 rounds, 80% of the n=50-asymptote precision arrives at n=16, not 7), and EV5's drift components fail for diagnosable reasons (Section 6.3: degenerate within-cell sandwich variance; PSI saturation at 5 obs/half). Meanwhile EV2 replicates 37/39 with max absolute error 0.038 on a five-fold extrapolation to 50 rounds (C1).
G-theory converts iteration counts into a solvable design problem
Actionable practitioner deliverables
Weaknesses
Provenance circularity plus commercial conflict of interest
Unexplained observation-count discrepancy
Headline power analysis admitted to be mis-specified at the binding level
Metric-structure evidence from one narrow subset
Reliability estimates rest on small k with wide intervals
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
This is a meta-methodology paper: it formalizes the 'Dice Roll Method' (repeated identical prompts to LLMs, aggregated for brand-recommendation audits) into a standardized protocol comprising a variance decomposition (Eq. 3: token, prompt-phrasing, model, version/time, interactions), an inferential stack (NB-GLMM, Cliff's delta, block/parametric bootstrap, simr power, G-theory), and iteration-count guidance across five RQs (power, metrics, reliability, convergence, cost). Headline claims: three iteration tiers (n=5/10/15 giving G=0.58/0.74/0.81), a complementary four-family metric battery reduced to a three-metric core, and a pre-registered external validation replicating D-study predictions in 37/39 cells while showing the fixed tiers do not transfer — recast as 'pilot-then-solve'. Section 1 discloses up front that every reanalysed dataset is from the author's own prior studies and that the protocol ships in the author's commercial product (Rankfor.AI).
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 4 strengths & 5 weaknesses, analyzed in detail
- 5 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
How many times should you repeat the same prompt when auditing LLM brand recommendations?
Per this paper: run a roughly 10-iteration pilot, estimate the variance components, and solve for a reliability target G >= 0.80. On the authors' own data, 5 iterations give G=0.58, 10 give G=0.74, and 15 cross the 0.80 group-level threshold (G=0.81). Because fixed iteration counts failed to transfer to external datasets, the 'pilot-then-solve' rule — not any fixed number — is the actual prescription.
What is the Dice Roll Method?
A protocol that issues the same prompt to an LLM many times and aggregates the brand recommendations, treating each query as one roll of a die sampled from the model's output distribution. The paper standardizes iteration counts, stability metrics (count-based, set-based, embedding-based, and fairness-adjusted PASOR), reliability targets via generalizability theory, and drift diagnostics.
Was the method validated on data the authors did not collect?
Partially. A pre-registered validation on three independent public corpora (Motoki's 100-round political-bias data, Rozado's 24-model sweep, and the llm-stability benchmark) replicated the reliability-prediction machinery in 37 of 39 cells and confirmed non-determinism at temperature 0. However, the fixed iteration tiers did not transfer, and external validation within the brand-recommendation domain itself remains open.
Does this paper have a conflict of interest?
Yes, and it is disclosed: the author is CEO of Rankfor.AI, which implements the protocol in a commercial product, and all five reanalysed datasets come from the author's own prior studies. The declarations state the company had no influence on the research, but the core data is only available 'on reasonable request'.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- The provided text is a sampled excerpt: several middle stretches fall inside '[...sampling gap...]' markers (parts of Sections 3, 5.1-5.2, 5.9-5.11, 6.1, 7.4-7.6 and appendices); judgments about content visible only there were avoided.
- Table 2's cells are not visible in the excerpt (only Table 3 fragments and the text's description of Table 2 as saturating); assessment of the pooled power table relies on the paper's own textual characterization.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.