Rui Yang

Research profile

Research summary

Rui Yang is an AI safety and security researcher affiliated with Huawei Technologies whose recent work concentrates on evaluating and hardening large language model (LLM) systems. The cataloged record is entirely from 2026 and comprises four arXiv preprints, three of them first-authored. Two threads dominate. The first is evaluation infrastructure: a 37,660-pair Chinese safety QA benchmark that stress-tests 4 LLMs and 7 safety judge models, and a paired-context study quantifying how conversational framing shifts cybersecurity-assistance compliance by +23.1 and −45.1 percentage points. The second is security analysis, notably a systematization of knowledge mapping 197 multi-agent LLM security papers through an A→I→R framework with a 44-work benchmark audit. A middle-authorship paper, HarnessEvolve, extends this agenda to agent self-evolution, reporting gains over GEPA, ACE, and SkillOpt on 5 benchmarks. OpenAlex topic data mirrors this spread: adversarial robustness in machine learning (3 works), security and verification in computing (2), and hate speech and cyberbullying detection (2). Bibliometric signals are early-stage: 5 works in the last three years, 26 distinct coauthors, h-index and i10-index recorded as 0, with radar scores of 66 for collaboration and 52 for activity, while output, impact, focus, and reproducibility remain unscored pending sufficient data.

Research keywords

Highlights

• C-SafeQA: a 37,660-pair Chinese safety QA benchmark auditing LLM judges

  First-authored benchmark of 37,660 Chinese QA pairs that stress-tests 4 LLMs and 7 safety judge models. Evolve's verdict: useful, but model-derived labels and the absence of a benign set temper its audit authority. Reproducibility readiness: medium.

• SoK systematizes 197 papers on multi-agent LLM security

  First-authored systematization mapping 197 works on multi-agent LLM security through an A→I→R framework, plus an audit of 44 benchmarks. Evolve calls the lens rigorous but notes the coding validity is unverified. Reproducibility readiness: low.

• Conversational context swings cybersecurity assistance compliance by tens of points

  First-authored paired evaluation showing accepted precedent raises compliance by 23.1 percentage points while request decomposition cuts it by 45.1. Evolve finds the paired design sound but notes labels rest on automated coders with no human gold check. Readiness: low.

• HarnessEvolve evolves agent harnesses from reference trajectories

  Middle-authorship work introducing reference-trajectory learning with dual gates for agent self-evolution, reported to beat GEPA, ACE, and SkillOpt on 5 benchmarks — though Evolve notes the absence of significance tests. Reproducibility readiness: low.

Timeline

1. 2026First-authored "Who Judges the Judges?", a 37,660-pair Chinese safety QA benchmark evaluating 4 LLMs and 7 safety judges.
2. 2026First-authored SoK mapping 197 multi-agent LLM security papers through an A→I→R framework, including a 44-work benchmark audit.
3. 2026First-authored paired evaluation of LLM cybersecurity assistance across conversational contexts (compliance +23.1pp / −45.1pp).
4. 2026Co-authored (middle position) HarnessEvolve on reference-trajectory learning for reliable agent self-evolution.

Papers in the Evolve catalog

• 2026-09-01first author

  Who Judges the Judges? A Chinese Safety QA Benchmark for Evaluating LLM Responses and Safety Judges

• 2026-09-01

  HarnessEvolve: Learning from Reference Trajectories for Reliable Agent Self-Evolution

• 2026-09-01first author

  SoK: When Safe Agents Fail Together: The Security of Multi Agent LLM Systems

• 2026-09-01first author

  Same Request, Different Boundary: Evaluating Cybersecurity Assistance across Conversational Contexts

Frequent co-authors