Order-Invariant Answers, Order-Sensitive Representations in Mathematical Reasoning
cs.LG · cs.AI · cs.CL · cs.SC
Quick verdict
Across 16 open models, permutation SNR of hidden states correlates with accuracy (Spearman up to 0.862), but small, ancestry-overlapping sample and unaddressed pooling confounds keep it a correlational probe.
Summary
The paper asks whether accurate reasoning on an answer-invariant task forces invariant internal representations. It builds a synthetic function-composition task: (m+1)D rule lines, where the queried chain plus m distractor chains are shuffled under a fixed bank of 50 permutation patterns, guaranteeing the answer stays fixed. Permutation SNR compares separation among the 50 ordering-group means of mean-pooled prompt hidden states against within-group variation, averaged over all layers. Across 16 instruction-tuned models (1B-8B; Qwen2.5, Llama-3.x, Mistral, Gemma-3, AceInstruct/AceMath) and three settings (D,m) of (2,4), (2,7), (3,4) with T=100 instances, layer-averaged SNR correlates positively with accuracy: Spearman rho = 0.862, 0.729, 0.621 (p = 1.78e-05, 1.34e-03, 1.03e-02). Partial correlations controlling model size remain positive (0.815, 0.647, 0.469) but the three-step setting drops below significance (p = 0.078). The central message is conceptual: answer invariance does not imply representation invariance, since more accurate models separate orderings more distinctly. Key limitations, acknowledged by the authors: purely correlational evidence from only 16 partly ancestrally overlapping models, no code or data release, unreported per-layer profiles, and answer-extraction details in an appendix absent from the text.
Review complete
Assessment at a glance
4
risk flags identified
● high 1
● medium 2
● low 1
5 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- Reordering the rule lines of the synthetic function-composition problems preserves the correct answer, making the task answer-invariant by construction (Sections 2.1, B.2).
- Permutation SNR — separation of the 50 ordering-group means of mean-pooled hidden states relative to within-group variation, averaged over layers — is higher in models with higher accuracy (Section 3.2).
- Layer-averaged permutation SNR is positively rank-correlated with accuracy in all three settings, with Spearman rho of 0.862 (2,4), 0.729 (2,7), and 0.621 (3,4) (Figure 1).
- The correlation persists after controlling for model size via partial Spearman correlations (0.815, 0.647, 0.469), though the three-step setting is inconclusive (p = 0.078) (Appendix C, Table 1).
- Answer invariance and representation invariance are distinct: more accurate models represent rule orderings more distinctly internally (Sections 1, 4).
Strengths
Cleanly controlled task design ensuring answer invariance by construction
Section 2.1/B.2: distinct values at each chain position guarantee a unique relevant chain; a single permutation-pattern bank of 50 orderings is sampled once per setting and applied to all T=100 instances, so ordering effects are not confounded with instance-specific patterns; answers are identical across all 50 presentations of a problem.
Transparent, reimplementable representation statistic
Section 2.2 gives the full formula: between-cluster variation of the 50 permutation-group means over within-cluster variation (+epsilon = 1e-8), mean-pooled over non-padding prompt tokens, then averaged over the embedding layer and all transformer layers. Public checkpoints and explicit prompt templates (B.3) make the pipeline replicable.
Model-size confound is explicitly addressed
Broad and well-documented model pool
Candid limitations and scoping
Weaknesses
Mean-pooled representations invite a position/token-position confound
Small, ancestry-overlapping model sample limits the cross-model inference
No multiple-comparison handling and no uncertainty on correlations
Accuracy distribution and per-ordering variability are never reported numerically
No code or data availability statement and missing appendix details
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
The paper (arXiv:2609.28442v1, Princeton; cs.LG primary) asks a representational question: when a model answers a reordering-invariant task correctly, do its internal representations stay invariant too? It constructs a synthetic multi-step function-composition task (Section 2.1): (m+1) chains of D numerical mappings plus distractors, shuffled under a shared bank of 50 permutation patterns, guaranteeing answer invariance by construction. The quantitative tool is permutation SNR (Section 2.2): separation among the 50 ordering-group means of mean-pooled prompt hidden states relative to within-group variation, averaged over all layers. Three claims anchor the paper: (i) the task isolates presentation order as the only manipulated variable; (ii) layer-averaged permutation SNR is positively rank-correlated with accuracy in all three settings — Spearman rho = 0.862 (D,m)=(2,4), 0.729 (2,7), 0.621 (3,4), each with p <= 1.03e-02 (Figure 1); (iii) answer invariance and representation invariance are distinct concepts. All three claims are supported by the reported evidence, with the caveats below.
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 5 strengths & 5 weaknesses, analyzed in detail
- 4 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What is permutation SNR in this paper?
It is a statistic computed from a model's hidden states. The same 100 problems are each presented under 50 fixed rule orderings; the statistic compares how far apart the 50 ordering-group average representations are (between-cluster variation) relative to how much representations vary among problems sharing one ordering (within-cluster variation), then averages this ratio over all model layers. High SNR means the model's pooled prompt representations separate rule orderings distinctly.
What is the main finding?
Across 16 open instruction-tuned language models (1B-8B parameters), models that solve the reordered function-composition problems more accurately also represent different rule orderings more distinctly in their hidden states. The Spearman correlation between accuracy and layer-averaged permutation SNR was positive in all three task settings, strongest at 0.862, though it weakened to 0.469 (not significant at p = 0.078) in the hardest setting after controlling for model size.
Does this mean encoding rule order helps reasoning?
No. The paper is explicitly correlational: it shows an association between accuracy and order-sensitive representations across models, not that order encoding causes better reasoning. The authors state that interventions on ordering-related representations are needed to test causality.
Is the code or data available?
The extracted text contains no code or data availability statement. The setup is described in enough detail (model list, prompt templates, problem construction, SNR formula) to be reimplemented from public checkpoints, but no official release is mentioned.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Figures are not analyzable from text extraction: Figure 1 is the only visualization of the headline correlations, and no per-model accuracy numbers appear in the text.
- The decoding-settings and answer-extraction details referenced to Appendix B are not present in the provided text; accuracy measurement quality cannot be fully audited.
- Any per-layer SNR profiles beyond the layer-averaged value are unreported or missing from the extraction, so layer-wise robustness cannot be judged.
- Only the main claim chain (task, metric, three correlations, size-adjusted analysis) could be verified against the extracted text.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.