GYROval: A Robust Benchmark for Cultural Value Orientation in Large Language Models
Alexander Didenko, Anna Shabanova, Vladislav Zapylikhin, Alexander Antipov, Ruslana Raemgulova
cs.HC · cs.AI · cs.CY
Quick verdict
GYROval: LLM value rankings survive role, theme, language and temperature; strong stats, but uncounterbalanced option order and no human baseline temper claims.
Summary
GYROval is a 224-item benchmark (one duplicate row acknowledged) measuring LLM cultural value orientation on the two Inglehart-Welzel axes via binary contrastive vignettes, fully crossing seven themes, four decision modalities, and eight value facets. Twenty models were administered the English instrument at temperature 0 with 30-50 samples per item; eleven also received a paired Russian translation and a vendor-recommended second temperature. Stability is assessed by ranking models within levels of each perturbation factor and comparing tie-corrected Kendall's W against 20,000-replicate permutation nulls, with vignette-stratified bootstrap intervals, a 48-specification sensitivity grid, and empirically derived stability ceilings. All twelve (factor, axis) cells exceed the permutation null; temperature and language preserve rankings best, decision role worst (modality on the sacred-secular axis reaches only 62% of its perfect-stability ceiling). Absolute scores shift by more than 20% of the scale across roles and themes, and per-model role sensitivity varies 6.5-fold. A mechanistic sub-study on five open-weight models shows value axes are linearly decodable and causally patchable, but cross-model direction alignment is mostly at chance. Key limitations: no human baseline or external criterion, a single prompt template, option order not counterbalanced despite a documented 91.5% counted-pole-in-Option-2 asymmetry, LLM-generated items with undisclosed expert criteria, and effectively 19 distinct models since Alice AI LLM duplicates YandexGPT. The instrument and audit scripts are public; raw model responses are not.
Review complete
Assessment at a glance
6
risk flags identified
● high 1
● medium 3
● low 2
4 strengths · 5 weaknesses · 5 author questions
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- Model rank order on the two Inglehart-Welzel axes persists across decision role, thematic domain, language of administration, and sampling temperature, with all twelve (factor, axis) concordance cells exceeding a 20,000-replicate empirical permutation null.
- Absolute counted-pole levels shift by more than 20% of the scale across decision roles (range 0.224) and themes (range 0.207), so benchmark scores are uninterpretable without specifying item composition.
- Role sensitivity is an intrinsic, widely varying model property: the score range across the four decision modalities spans 0.071-0.461 on the emancipative axis and 0.094-0.500 on the sacred-secular axis.
- Decision role is the strongest perturbation (modality/sacred-secular preserves only 62% of its perfect-stability ceiling), while temperature and language are nearly benign (0.966-0.989 of ceiling).
- On five open-weight models, value axes are linearly decodable from residual-stream activations (68.3-96.7% probe accuracy vs 51.7-90.0% baselines) and causally patchable within models, but cross-model value-direction alignment is mostly at chance.
Strengths
Unusually candid self-audit of the instrument
Section 2.1 documents its own duplicate row, four orthographic variants of the axis labels (93/19 and 88/25 splits), the counted pole residing in Option 2 for 91.5% of polarity-identifiable items, and systematic Option 2 length/hedging asymmetry (paired t=6.12 and t=2.44) - structural facts most benchmark papers omit; the audit script is released.
Statistical pipeline well above field norm
Vignette as unit of analysis with ICC 0.748-1.000 and design effects 22.7-50 (Section 3.1); tie-corrected Kendall's W against 20,000-replicate permutation nulls per cell (Table B.1); 48-specification sensitivity grid shifting W by <=0.0051 (Section 3.9); empirically derived stability ceilings via item rotation (Section 3.5).
Honest separation of rank stability from measurement invariance
Fully crossed design and paired bilingual protocol
Weaknesses
Fixed option order despite documented slot asymmetry
No human baseline or external criterion validity
Mechanistic claims outrun their item counts
Inconsistency between stated unit-of-analysis doctrine and reported inference
Panel integrity and composition
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
- All 4 strengths & 5 weaknesses, analyzed in detail
- 6 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What is GYROval and what does it measure?
GYROval is a publicly released benchmark of 224 short two-option scenarios that measures the cultural value orientation of large language models along the two Inglehart-Welzel axes (sacred vs secular, obedient vs emancipative). Neither option is correct; a model's score on an axis is simply the share of its choices falling on one designated pole, aggregated across scenarios spanning seven topics, four decision roles, and eight value facets.
Are LLM rankings on cultural-value benchmarks stable when the test conditions change?
In this study, model rankings stayed above chance under all four tested perturbations (decision role, topic, language, sampling temperature), but were not perfectly stable. Changing the sampling temperature or language barely reordered models, while changing the decision role moved mid-table ranks by up to 9-14 positions, and ranks on the sacred-secular axis reached only about 62% of the best achievable stability. Absolute scores also shifted by more than 20% of the scale across roles and topics.
Does asking a model in Russian instead of English change its cultural values?
Only slightly. Across eleven models tested with paired translations of identical items, Russian administration shifted ten of eleven models toward the emancipative pole by an average of 0.029 on a 0-1 scale, with no statistically significant rank reordering (pooled rank correlation between the English and Russian league tables was about 0.82). The shifts did not track whether the model's developer was Russian.
Is there a human baseline for the GYROval benchmark?
No. The paper explicitly states that no human participants were evaluated, no human performance ceiling was established, and no external criterion measures were gathered; the study therefore evaluates relative model ordering and rank stability rather than accuracy or calibration against human populations, and the authors list this as their primary limitation.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Figures (rank matrices, heatmaps, layer-depth profiles) cannot be analyzed from extracted text; captions note some graphic labels are in Russian, which the authors flag as a source-graphic artifact
- The extracted text may omit parts of the paper (e.g., any appendix beyond A and B); judgments rely on the provided full text only
- Claims about the released repository (audit reproducibility, excluded raw responses) are taken from the Data Availability section and were not verified against the actual repository
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.