Scaling Articulated Rationales for MLLM-based Recommendation
Haoke Xiao, Yueyang Liu, Yuhui Zhang, Xiang Chen, Yufei Liu, Jia Xu, Yalong Guan, Xiaolan Zhu, et al.
cs.IR · cs.AI
Quick verdict
SARA claims an industrial pipeline that turns sparse user-stated like/dislike reasons from Kuaishou Live into MLLM-generated rationale features with online gains, but its headline numbers lack significance testing and its evaluation judge is entangled with model training.
Summary
This industrial system paper introduces articulated user rationales (AURs): natural-language explanations of why a viewer likes or dislikes a livestream. SARA has three parts. (1) A questionnaire-based data engine over 240M Kuaishou Live users collects 1.92M candidate rationales in ~185 days; an Agent-as-a-Judge rubric (six dimensions, non-compensatory holistic score) retains 187.5K (9.8% yield) as SARA-HQ, organized into 86,564 author-centric polarity profiles. (2) Qwen2.5-VL-7B is fine-tuned on SARA-HQ and refined with Quality-Refining DPO using self-sampled candidates ranked by the offline Agent Judge, yielding SARA-7B, which generates rationales for 10M authors. (3) Positive rationales are integrated into an MMoE ranker via mutual-information-aligned UA interaction embeddings; negative rationales are residual-quantized into semantic IDs attached to dislike histories. On a 5K author-disjoint set, SARA-7B-DPO scores higher than Gemini-3.1-Pro and Qwen3-VL-8B (holistic 3.33 vs 2.51/2.37; BGE-Sim 0.82 vs 0.70/0.69). Online A/B tests on 1% traffic report click +0.34%, watch time +0.99%, dislikes -8.16% over 30+ days. Key limitations: no significance testing, an evaluator trained partly on SARA outputs also scores baselines, BGE-Sim is structurally biased toward SFT imitators, and all data/code remain proprietary.
Review complete
Assessment at a glance
6
risk flags identified
● high 2
● medium 3
● low 1
5 strengths · 5 weaknesses · 5 author questions
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- Articulated user rationales form a new, polarity-aware, reason-level textual signal that is practically usable in industrial recommendation after curation and scaling.
- SARA-7B (Qwen2.5-VL-7B + SFT + Quality-Refining DPO) generalizes rationale generation from 86,564 covered authors to the full 10M-author space, outperforming Gemini-3.1-Pro and Qwen3-VL-8B on a 5K author-disjoint evaluation set.
- Integrating generated positive rationales into ranking via MIM-aligned UA interaction modeling yields consistent offline AUC/GAUC gains and online engagement improvements (e.g., watch time +0.986%).
- Attaching negative-rationale semantic IDs to dislike histories reduces online negative feedback (Hate -8.16%) rather than merely redistributing it.
- The unified six-dimensional, non-compensatory judge framework is validated against expert annotation, approaching inter-expert agreement (holistic MAE 0.20 vs 0.12 upper bound).
Strengths
Genuinely novel supervision signal at industrial scale
Table 2 convincingly positions SARA-HQ as the first dataset where a reason-level, polarity-aware, open-ended human explanation is the direct supervision unit (187.5K instances from real users), unlike LiveRec/KuaiLive (implicit logs) or LiViBench/LiveCC (content understanding without preference grounding).
Transparent quantity-quality pipeline accounting
Sec. 2 reports full numbers end to end: 1,915,718 candidates -> 187,532 HQ (9.8% yield), polarity split (positive 6.8% vs negative 24.2% retention), 82% of responses under ten Chinese characters, and the UA matrix sparsity (141,461 x 86,564; 1.33 responses/user, 2.17/author), which motivates author-centric aggregation.
Validated, non-compensatory evaluation protocol
Both integration routes ablated and deployed
Self-aware handling of the post-hoc rationalization risk (R3)
Weaknesses
No statistical rigor for the headline results
Evaluation judge is entangled with the system being evaluated
Faithfulness metric structurally favors the fine-tuned model
Generated rationales attached to real user dislikes inherit the R3 problem
No fairness, ethics, or privacy discussion
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
- All 5 strengths & 5 weaknesses, analyzed in detail
- 6 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What is an articulated user rationale (AUR) in SARA?
It is a short natural-language statement from a real user explaining why they liked or disliked a livestream, collected via a two-step questionnaire (first choose positive/negative, then state the reason). SARA treats these reasons as training supervision, unlike typical recommenders that only use clicks, watch time, and skips.
How does SARA-7B differ from using GPT or Gemini directly for generating user preference explanations?
SARA-7B is Qwen2.5-VL-7B fine-tuned on 187,532 human-written, quality-filtered preference rationales from Kuaishou Live, then refined with DPO. On the authors' evaluation set it produces more specific and better-grounded rationales than Gemini-3.1-Pro or Qwen3-VL-8B, though the comparison judge was trained by the same team.
What online improvements did SARA achieve on Kuaishou Live?
In A/B tests on 1% of production traffic, deployed for over 30 days with daily refresh: click +0.342%, watch time +0.986%, effective views +0.616%, follows +0.616%, and hate (dislike) actions reduced by 8.16%. No statistical significance values are reported in the paper.
Is the SARA-HQ dataset or SARA-7B code publicly available?
No. The paper reports no code or data release; SARA-HQ comes from proprietary Kuaishou Live questionnaires, so independent replication is not currently possible.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Text was extracted from PDF: figures (Figs. 1, 2, 4-14) are unavailable as images and were judged only from captions and surrounding text; axis details and qualitative-example screenshots could not be verified.
- Some citation strings look garbled by extraction (e.g., 'Ruochen Yang and Yueyang', 'LiveStar' listed as 'OmniStar' in Table 2 context); reference details were not verified.
- Appendix rubric (Table 8) was present but appendix figures and any omitted sections (e.g., full hyperparameters, training compute) are absent from the extraction, so reproducibility judgments rest on reported descriptions only.
- Online A/B configuration details (randomization unit, duration per experiment, guardrail metrics) are not fully described in the extracted text.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.