arXiv paper2026-09-02· recommender systems (evaluation methodology)· empirical study

Training seeds and model-selection stability in recommender-system evaluation

Juan Manuel Rodriguez, Oleg Lesota, Antonela Tommasel

cs.IR · cs.LG

Quick verdict

Short study: training-seed variation alone shifts metrics, model selection and top-k lists in RecSys eval; convincing but effect sizes thinly reported.

Summary

The paper asks whether changing only the training seed—holding the data partition, hyperparameters and evaluation protocol fixed—alters recommender-system evaluation conclusions. Using RecBole, the authors train BPR (3 configurations), NeuMF (9), BERT4Rec (12) and SASRec (12) on MovieLens-1M, Steam and Amazon All Beauty 2023, repeating every model–configuration pair with seeds 1–10 (chosen a priori), up to 300 epochs with early stopping (patience 10) and best-validation checkpointing; nDCG@10 is the primary metric. Four analyses are run: (1) Friedman tests over per-user nDCG distributions across seeds find seed variation often detectable—all configurations rejected on Steam; on ML-1M clearly for BERT4Rec/SASRec, partially for NeuMF, weakly for BPR; (2) validation-based configuration selection is quantified as relative regret (%) across selection/evaluation seeds, showing low-loss instability (NeuMF/Steam) but large validation-to-test gaps (NeuMF, BERT4Rec on Amazon); (3) validation pairwise wins transfer inconsistently to test (weakest on Amazon, strongest on Steam); (4) cross-seed top-k list agreement (Jaccard and Average Overlap@k) is dataset-dependent, from near-disjoint BPR lists on Amazon to high agreement on Steam, always below 1. Stated limitations: no attribution of variance to specific stochastic mechanisms, one fixed data partition, three datasets, four models, accuracy metrics only.

Review complete

Assessment at a glance

recommender systems (evaluation methodology)

risk flags identified

● high 0

● medium 2

● low 1

Reproducibility readinesshigh

Code & data availableyes

Review confidencehigh

AI text signal0/17 passages flagged

Deep research viableCode-reproducibleNeeds GPU

4 strengths · 4 weaknesses · 5 author questions

AI text signal = Binoculars zero-shot detector score over 17 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.

Levels reflect verification complexity — not a quality score. Evolve never rates your paper.

Key claims under review

With the data partition fixed, varying only the training seed produces detectable differences in per-user nDCG@10 distributions for most model–dataset combinations (Friedman test; all configurations rejected on Steam).
Seed variation can change the validation-selected hyperparameter configuration and create validation→test performance gaps, with impact governed by configuration separation and validation–test transfer quality.
Different seeds never produce identical top-k lists; agreement ranges from near-disjoint (BPR on Amazon All Beauty) to high (Steam), with Average Overlap@k consistently above Jaccard@k.
Reporting single-seed results can overstate evaluation stability, so training seeds should be treated as part of the evaluation protocol rather than implementation noise.

Strengths

Clean isolation design
Fixes the data partition and varies only the training seed across 3–12 hyperparameter configurations per model (§2), with the seed set {1,…,10} fixed a priori to avoid seed tuning — a rigorous control much prior seed-variance work lacks
Multi-level, decision-relevant framing
Separates statistical detectability (Friedman per-user tests) from practical consequences (validation-selection regret %, val→test win transfer, list agreement), correctly distinguishing e.g. NeuMF/Steam (detectable but low regret) from BPR/Amazon (stable selection yet poor transfer) in §3 and Fig 2
Goes beyond aggregate metrics
Strong reproducibility posture

Weaknesses

Statistical reporting is minimal
Single fixed data partition
Inline evidence covers only subsets
Configuration gaps rest on RecBole defaults

Full Review Report

$1.99 · one-time

Unlock the complete preliminary analysis Evolve produced for this paper.

Report preview

Scope & Claims

This 4-page RecSys '26 short paper asks a deliberately narrow question: with the data partition, hyperparameter space and evaluation protocol held fixed, does varying only the training seed change the conclusions a researcher would draw? The authors define the seed as controlling the full training run (initialization, mini-batch order, dropout, masking, latent and negative sampling) and explicitly disclaim attribution to any single mechanism (§1, §4). The claims are calibrated: seed variation is "often detectable"; its practical meaning depends on configuration separation, validation→test transfer, and list agreement (Abstract, §4). The normative recommendation — treat seeds as part of the evaluation protocol — follows from the evidence rather than outrunning it. This restraint is a strength; few reproducibility papers resist overclaiming.

Methodology Assessment

The design is a clean factorial isolation. Three datasets (MovieLens-1M, Steam, Amazon All Beauty 2023) receive per-user temporal 80%/10%/10% splits. Four models span MF and sequential-neural families: BPR (3 embedding sizes), NeuMF (9 MF×MLP combinations), BERT4Rec and SASRec (12 configurations each). Every model–configuration pair is retrained under seeds {1,…,10}, fixed a priori to preclude seed tuning; training runs up to 300 epochs with patience-10 early stopping and best-validation checkpointing; remaining hyperparameters use RecBole defaults (§2). Three choice

The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.

All 4 strengths & 4 weaknesses, analyzed in detail
3 risk flags with severity, section & reviewer questions
Methods & statistical evidence assessment
5 questions a peer reviewer would ask
Full six-dimension evaluation report & recommendations

One-time purchase · this paper only

One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.

Frequently asked questions

Does changing the random seed change recommender system evaluation results?

According to this study, yes: with the data split and hyperparameters fixed, retraining with each of 10 different seeds produced statistically detectable differences in per-user nDCG@10 for most model–dataset combinations (all configurations on Steam), and could change which hyperparameter configuration validation selects.

Which datasets, models and seeds were used in the study?

MovieLens-1M, Steam and Amazon All Beauty 2023, with four models (BPR, NeuMF, BERT4Rec, SASRec) implemented in RecBole, each trained with seeds 1–10 across 3–12 hyperparameter configurations, up to 300 epochs with early stopping (patience 10).

Do different training seeds recommend different items to users?

Yes — top-10 lists were never identical across seeds. Agreement ranged from nearly disjoint lists (BPR on Amazon All Beauty) to high agreement (most models on Steam), and Average Overlap@10 was consistently higher than Jaccard similarity, meaning agreement is better near the top of the ranking.

Is the code and data for this paper available?

Yes. The paper links a companion repository (github.com/knife982000/RecSeed) containing code, full hyperparameter spaces, results, and additional metrics (Recall@10, MRR@10, Hit@10, Precision@10). All datasets used are public.

About this review

This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.

Figures appear in the extraction only as axis labels, legends and captions; exact plotted values (regret percentages, Jaccard/AO values, boxplot distributions) could not be read, so figure-level numbers in this review are limited to visible axis ranges (e.g., 0–40% regret for NeuMF/Amazon vs 0–4% for NeuMF/Steam in Fig 2).

Go deeper on this paper

Wrote a paper? Stress-test it before reviewers do.

Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.

Scan your paper