SALA: Semantic-Aware Logical Alignment for Complex Reasoning in In-Context Learning
Zhao Ji, Wenqing Chen, Zhixuan Chu, Jianxing Yu, Jingping Liu, Shanhe Zhao, Zibin Zheng
cs.AI · cs.CL
Quick verdict
SALA adds LLM-induced operations + DTW alignment to demo selection, topping 4 benchmarks × 3 LLMs on average; wins lack significance testing.
Summary
The paper proposes SALA, a demonstration-selection framework for in-context learning aimed at complex reasoning. It replaces PSL's fixed inventory of 13 QDMR operations and exact prefix matching with two mechanisms: an LLM (Seed-OSS-36B-Instruct) induces task-specific reasoning operations from each benchmark's training set, deduplicated by normalized-name heuristics and an LLM overlap judgment; and operation descriptions are embedded with bert-base-uncased, after which dynamic time warping aligns the query's and candidates' operation-embedding sequences, yielding a length-normalized similarity used for top-k=8 retrieval with easy-to-hard ordering. Across GSM8K, SVAMP, CommonsenseQA and StrategyQA with Llama3-8B-Instruct, Qwen2.5-7B-Instruct and DeepSeek-V4-Pro, SALA attains the best average accuracy (82.76, 85.23 and 91.42 respectively), and on DeepSeek it wins all four benchmarks, though it loses GSM8K to LMS3 on the two open-weight models (82.59 vs 82.65; 91.16 vs 91.52). Ablations attribute average drops of 0.5-0.6 points (Llama3) and 1.5-1.8 points (Qwen2.5) to removing either module, and over 68% of samples (over 98% on GSM8K) use induced operations. Main limitations: results are five-run means without variance or significance tests; answers are judged by an unspecified GPT-4 rather than standard metrics; EPR and LMS3 are absent from the DeepSeek comparison; and no code release is stated.
Review complete
Assessment at a glance
6
risk flags identified
● high 0
● medium 3
● low 3
5 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- Inducing task-specific operations beyond the 13 predefined QDMR operations improves reasoning demonstration selection (average ablation drops of 0.5 points on Llama3-8B and 1.5 on Qwen2.5-7B when removed).
- DTW-based semantic alignment outperforms exact prefix matching (average drops of 0.6 and 1.8 points on the two open models when replaced by prefix matching).
- SALA achieves the best average accuracy across four reasoning benchmarks on all three LLMs: 82.76 (Llama3-8B-Instruct), 85.23 (Qwen2.5-7B-Instruct) and 91.42 (DeepSeek-V4-Pro).
- Task-specific operations are widely needed: over 68% of samples use them (over 98% on GSM8K) in both training and test splits.
Strengths
Genuine, well-motivated advance over PSL
Section 1-2 clearly diagnose the two failure modes of PSL (fixed 13-operation inventory, exact prefix matching); SALA's task-adaptive operation space (3.2) and DTW alignment (3.4) address both, and the case studies (Tables 7-10) show concrete failures of prefix matching, e.g. the best structural match excluded because c3 ≠ q3 (Table 9).
Fully documented, reproducible-in-principle pipeline
All five prompts (C.1-C.5), hyperparameters (Table 5: top-k=8, temperature 0.01, bert-base-uncased 768-d embeddings), and a deterministic two-stage deduplication procedure (3.2.2, normalized-name containment + LLM binary judgment, Eq. 3) are specified.
Ablations cleanly isolate both modules and are internally consistent
Quantified evidence for operation-induction necessity
Broad evaluation grid
Weaknesses
No variance, CIs, or significance tests
Nonstandard LLM-judge evaluation
Closest baselines missing on the strongest model
Related reasoning-aware methods discussed but not run
No efficiency analysis and incomplete artifact release
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
SALA (arXiv 2609.02336) targets demonstration selection for in-context learning (ICL) on complex reasoning. Its positioning: surface-similarity retrievers miss problem-solving logic, while the closest prior method, PSL (Ma et al., 2025), matches reasoning at the operation level but with a fixed inventory of 13 QDMR operations and rigid exact-prefix matching. SALA claims three contributions: (1) a task-adaptive operation space induced from downstream training data; (2) DTW-based semantic alignment over operation-embedding sequences; (3) consistent average gains on four benchmarks (GSM8K, SVAMP, CommonsenseQA, StrategyQA) across three LLMs (Llama3-8B-Instruct, Qwen2.5-7B-Instruct, DeepSeek-V4-Pro), evaluated on the splits in Table 3 (e.g. GSM8K 7,473/1,319).
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 5 strengths & 5 weaknesses, analyzed in detail
- 6 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What is SALA and what problem does it solve?
SALA (Semantic-Aware Logical Alignment) is a method for choosing which demonstration examples to include in an LLM prompt (in-context learning). Instead of retrieving examples that are merely worded similarly to the question, it matches the underlying problem-solving logic: questions are decomposed into sequences of reasoning operations, and demonstrations whose operation sequences align best with the query are selected, improving accuracy on reasoning benchmarks like GSM8K.
How is SALA different from PSL?
PSL, the closest prior method, represents problems with a fixed set of 13 QDMR reasoning operations and selects demonstrations via exact prefix matching of these operation sequences. SALA extends the fixed inventory with operations automatically induced from each task's training data by an LLM, and replaces exact matching with dynamic time warping (DTW) over operation embeddings, allowing flexible, semantically soft alignment between sequences of different lengths and granularities.
What accuracy does SALA achieve?
Across four benchmarks (GSM8K, SVAMP, CommonsenseQA, StrategyQA), SALA achieves the best average accuracy of the compared methods on all three tested LLMs: 82.76 with Llama3-8B-Instruct, 85.23 with Qwen2.5-7B-Instruct, and 91.42 with DeepSeek-V4-Pro. Note it does not win every single benchmark — for example, LMS3 scores higher on GSM8K with the two open-weight models.
Is SALA's code or data publicly available?
The paper does not state a code or data release. It does document the full prompt templates (Appendix C), hyperparameters (Appendix D, e.g. top-k=8, bert-base-uncased embeddings), and example induced operations (Appendix E), but the complete induced operation sets per benchmark are not provided, so exact replication is currently not straightforward.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- PDF extraction interleaves table cells, figure labels and body text; all quoted numbers were cross-checked between Table 1/Table 2 and the ablation narrative, and the text ends cleanly after Table 11 with no truncation detected.
- Figures 2-4 are present only as flattened text in the extraction; values cited from them (e.g. Figure 3 percentages, Figure 4 sequence lengths) come from the embedded numeric labels, not from visual inspection.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.