SEEK: Skill-Routed Evaluation with Evolvable Knowledge for Industrial Search
Zhongxin Huang, Songyang Li, Renzhe Zhou, Feiran Zhu, Chenglei Dai, Zhen Xiao, Xuanping Li, Jingwei Zhuo
cs.IR · cs.AI
Quick verdict
Kuaishou's SEEK decouples evaluation skills from an 8B judge; strong ablations and production gains, but all data is private and no significance tests.
Summary
SEEK is an industrial framework from Kuaishou for automatic search-quality evaluation that decouples evaluation knowledge from the judging model. A bank of 12 diagnostic skills across five dimensions (relevance, quality, diversity, authority, heterogeneous content) stores explicit criteria as routing descriptions plus operational guidance; a Qwen3-0.6B router activates roughly 3–4 relevant skills per query–result-list pair, and a Qwen3-8B listwise evaluator produces page-level good/fair/bad labels with evidence-grounded attribution. Supervision comes from a Qwen3-235B teacher that converts 169,434 human-annotated Kuaishou pairs into routing targets and evaluation trajectories, followed by DAPO reinforcement learning with a hierarchical reward. On a 17,000-pair held-out set, SEEK reaches 0.6555 three-class Macro-F1 versus 0.6418 for same-backbone SFT+GRPO and slightly beats GPT-5.6 Sol on binary accuracy (0.7520 vs 0.7451), with the best attribution recall on all five dimensions per Table 1. A replay-gated skill-evolution loop absorbs recurring knowledge gaps without retraining, recovering most adaptation gains (within 0.0036–0.0041 F1 of model updates) with near-zero forgetting. In production on 24,000 pairs it scores 0.842 accuracy against expert-calibrated labels in 0.9 hours — a 40.6× speedup over outsourced review. Main limitations: entirely proprietary data, no significance testing, recall-only attribution reporting, an unexplained Fair-class F1 regression, and unquantified teacher-model circularity.
Review complete
Assessment at a glance
5
risk flags identified
● high 0
● medium 4
● low 1
5 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- Externalizing evaluation criteria into a 12-skill bank and dynamically routing 3-4 skills per query-list pair outperforms both exposing all skills and parameter-internalizing post-training (Tables 1, 2, 5).
- Two-stage SFT plus DAPO with a hierarchical reward aligns a Qwen3-8B listwise evaluator with human judgments better than SFT+DPO or SFT+GRPO (Tables 1, 3, 4).
- Replay-gated skill-only evolution recovers most adaptation gains (within 0.0036/0.0041 F1 of model fine-tuning) with near-zero forgetting (0.0008/0.0005) without touching model parameters (Figure 3, Table 6).
- In production, SEEK exceeds outsourced (0.783) and multi-round human review (0.830) accuracy against expert-calibrated labels (0.842) at 40.6× speed on 24,000 pairs (Table 7).
Strengths
Well-isolated component ablations
Table 2 disentangles routing vs evaluator training (binary accuracy 0.4894 → 0.6485 → 0.7520); Tables 3-4 ablate SFT/GRPO/DAPO and each reward term; Table 5 adds an oracle-routing upper bound (0.768) — unusual rigor for an industrial paper.
Distinctive frozen-model adaptation result
Figure 3 and Table 6 show skill-only evolution recovers adaptation to within 0.0036/0.0041 F1 of model fine-tuning with 0.0008/0.0005 replay forgetting vs 0.0076/0.0084 for parameter updates — directly quantifying the capability–knowledge separation.
Production-scale evidence with cost accounting
Documented annotation protocol and large corpus
Principled evolution guardrails
Weaknesses
No statistical significance testing
Teacher-model circularity
Recall-only attribution; unexplained Fair regression; table/text mismatch
Fully proprietary, unreproducible evaluation
Missing natural baseline for the core motivation
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
SEEK, from Kuaishou Technology and Peking University, addresses page-level quality evaluation for industrial short-video search. The paper's central claims: (1) externalizing evaluation criteria into a skill bank and routing a compact subset per query–list pair beats both packing all criteria into one prompt and internalizing them via post-training; (2) a two-stage SFT + DAPO pipeline over teacher-generated trajectories aligns a Qwen3-8B listwise evaluator with human page-level judgments and five-dimension attribution; (3) a replay-gated skill-evolution loop absorbs recurring knowledge gaps without retraining while preserving historical accuracy; (4) in production the system surpasses outsourced and multi-round human review on expert-calibrated accuracy at a fraction of the cost. Deployment at Kuaishou (400M+ DAU) is reported. All quantitative evidence comes from proprietary Kuaishou data: 169,434 training and 17,000 held-out query–list pairs.
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 5 strengths & 5 weaknesses, analyzed in detail
- 5 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What is SEEK and who built it?
SEEK (Skill-Routed Evaluation with Evolvable Knowledge) is a framework from Kuaishou Technology with Peking University for automatically evaluating search result quality at the page level. It keeps evaluation rules in an external skill bank of 12 diagnostic skills, routes 3-4 relevant skills per query, and uses a fine-tuned Qwen3-8B model to judge the whole result page. It is deployed in Kuaishou's production search evaluation pipeline, a platform with over 400 million daily active users.
How much faster and more accurate is SEEK than human review?
In Kuaishou's production study on 24,000 query-result pairs, SEEK completed the workload in 0.9 hours versus 36.5 hours for outsourced human review (a 40.6× speedup). Its judgments agreed with expert-calibrated reference labels more often than either outsourced review (0.842 vs 0.783 accuracy) or multi-round human review (0.830), though the expert-label construction process is not fully described in the paper.
Does SEEK need retraining when evaluation rules change?
Not immediately. New or changed evaluation knowledge is incorporated by revising the text of individual skills in the skill bank. Each revision is accepted only if it improves newly observed cases without degrading a frozen 5,000-pair historical replay set beyond a small tolerance. Measured forgetting after three update rounds was about 0.001 F1, versus roughly 0.008 when fine-tuning the model instead; only about 0.004 F1 of adaptation gain is lost compared to retraining.
Is the SEEK code, data, or skill bank publicly available?
No. The paper reports no code release, and all experiments use proprietary Kuaishou short-video search logs (169,434 training and 17,000 test query-list pairs). Only one example skill's text is shown in the appendices, so independent replication of the reported numbers is currently not possible.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Figures 1-3 are available only as garbled extraction; per-round values in Figure 3 and the labels in Figure 2 are partially unreliable, so adaptation-curve details were read with care.
- Page furniture shows unfilled ACM placeholders (ISBN XXXX, Conference 'XX), indicating a pre-camera-ready manuscript; minor inconsistencies may be fixed in the final version.
- References include 2026-dated works and models (GPT-5.6 Sol, Gemini 3.1 Pro, MiniMax-M2.5) that could not be independently verified from the text.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.