Iris: Climbing to the Search Frontier
Ziyuan Liu, Hengqi Liu, Zichuan Wang, Yang Qin, Jiachen Liang, Xu Chu, Shaowei Chen, Yuantao Gu, et al.
cs.AI
Quick verdict
Claims SOTA open-source search agents (BrowseComp 82.2/88.6) via synthetic multi-hop data + SFT-RL climbing; results credible but single-run, no error bars.
Summary
The AllSpark Team presents Iris-mini and Iris-pro, two search agents built from Qwen3.6-35B-A3B and Qwen3.5-397B-A17B. Training questions are reverse-constructed from web hyperlink structure: an entity graph is distilled from a seed page's out-links, multi-hop questions are authored over it, and every non-answer entity is rewritten into a descriptive reference so no clue resolves by string matching. Questions are kept only if a reference model fails them closed-book yet solves them given the evidence. Teacher trajectories pass trajectory-level filtering (correctness, compression-based degeneracy detection, minimum depth) and turn-level masking before SFT; RL against live search follows, with an in-house Qwen3.5-397B serving as reward judge and observation summarizer, request-level partial rollout with truncated importance sampling, and iterative SFT-RL 'climbing' that recycles the hardest solved and shortest successful rollouts into later supervised passes. The paper's distinctive contribution is evaluation hygiene: every benchmark is reported with and without inference-time context management (CM). With discard-all CM, Iris-mini reaches 82.2 on BrowseComp, 84.8 on BrowseComp-ZH, 86.9 F1 on DeepSearchQA, and 52.3 on HLE; Iris-pro reaches 88.6/85.1/92.9/56.4, claimed best among open systems at comparable scale. CM is worth up to +17.5 points (mini, BrowseComp; +21.2 with retry). Limitations: single pass@1 rollouts with no variance estimates or significance tests; most baselines imported from other technical reports under unknown judges; key hyperparameters and the evaluation judge undisclosed; DeepSearchQA for Iris-mini trails XYZ-Aquila-mini (86.9 vs 89.5).
Review complete
Assessment at a glance
6
risk flags identified
● high 1
● medium 3
● low 2
5 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- Multi-hop questions reverse-constructed from web hyperlinks, with all non-answer entities rewritten into descriptive references and admitted only when a reference model fails them closed-book yet solves them with evidence, yield training data strong enough for frontier-adjacent search agents.
- With discard-all context management, Iris-mini and Iris-pro achieve the strongest overall results among open-source search agents in their parameter ranges: 82.2/84.8/86.9/52.3 (BrowseComp/BrowseComp-ZH/DeepSearchQA/HLE) and 88.6/85.1/92.9/56.4.
- Inference-time context management is worth up to +21.2 points (Iris-mini, BrowseComp, discard-all+retry), so agent results must be reported both with and without it to separate policy from harness.
- Alternating SFT and RL (SFT-RL climbing), returning the hardest solved and shortest successful rollouts of each RL round to the next supervised pass, provides a self-paced curriculum that shifts toward harder questions as the policy improves.
- Search-specialized data and models transfer positively to general tool use (BFCL, tau-bench) and cowork domains (OfficeQA, APEX), suggesting search is an atomic capability rather than a vertical specialization.
Strengths
Dual-regime evaluation makes harness inflation measurable
Table 2 reports every configuration without CM first (Iris-mini 64.7 BrowseComp, 72.3 BrowseComp-ZH) and then under discard-all/retry, letting readers separate policy from harness; the paper explicitly argues many prior results are reported only under CM, obscuring this split
No-CM numbers provide the strongest evidence gains are not purely inference-time
Iris-mini at 64.7 BrowseComp beats FORT-Searcher (55.9), OpenSeeker-v2 (46.0), and REDSearcher (42.1) with no CM wrapper, and Section 4.3 explains CM gains via context-exhaustion frequency rather than benchmark headroom
Shortcut-resistant data pipeline with formal admission criteria
Candor about losses and benchmark defects
Turn-level filtering with data-induced rubric and principled RL engineering
Weaknesses
Single pass@1 evaluation with SOTA margins inside plausible noise
Baseline provenance undermines cross-system comparability
Recipe reproducibility deferred
Narrow contamination guard
Qualitative transfer claims without numbers
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
This technical report from the 'AllSpark Team' introduces two search agents - Iris-mini (35B-A3B, from Qwen3.6-35B-A3B) and Iris-pro (397B-A17B, from Qwen3.5-397B-A17B) - plus the data pipeline and SFT/RL recipe behind them. Headline claims: (1) questions reverse-constructed from web hyperlink structure, anchor-abstracted, and dual-verified (closed-book failure + evidence-conditioned solvability, Eqs. 5-9) produce training data strong enough for frontier-adjacent search; (2) with discard-all context management (CM), the models reach 82.2/84.8/86.9/52.3 and 88.6/85.1/92.9/56.4 on BrowseComp/BrowseComp-ZH/DeepSearchQA/HLE (Table 1), the best overall among open systems at their scales; (3) CM alone is worth up to +21.2 points (Iris-mini, BrowseComp, Table 2), so results must be reported with and without it. The paper is unusually explicit that benchmark scores conflate policy and harness.
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 5 strengths & 5 weaknesses, analyzed in detail
- 6 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What scores does Iris achieve on BrowseComp and the other benchmarks?
With the discard-all context-management setting, Iris-mini (35B) scores 82.2 on BrowseComp, 84.8 on BrowseComp-ZH, 86.9 F1 on DeepSearchQA, and 52.3 on HLE (text-only). Iris-pro (397B) scores 88.6, 85.1, 92.9, and 56.4 respectively. Without any context management, the scores drop to 64.7/72.3/81.0/43.2 for Iris-mini and 72.6/76.8/86.4/50.8 for Iris-pro.
How much does context management contribute to Iris's results?
A lot. For Iris-mini, the discard-all strategy adds +17.5 points on BrowseComp, +12.5 on BrowseComp-ZH, +5.9 on DeepSearchQA, and +9.1 on HLE over the no-context-management baseline; combining discard-all with retry raises BrowseComp to 85.9 (+21.2). Gains are smaller for the larger Iris-pro (+5.6 to +16.0), and the paper argues CM value tracks how often sessions run out of context.
Are the Iris model weights or code available?
Not yet at publication. The paper states the team plans to release the model weights together with key components of the data-construction, training, and evaluation pipelines via GitHub and HuggingFace links, but several details (hyperparameters, prompts, judge model) are deferred to 'future' release.
What base models is Iris built on?
Iris-mini is initialized from Qwen3.6-35B-A3B and Iris-pro from Qwen3.5-397B-A17B; both are mixture-of-experts models with a 256K-token context window. Training uses two epochs of SFT (batch 64, max sequence 262,144 tokens) followed by RL on the open-source Relax framework.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Figure 1 is extracted as flattened label/value text; the chart layout itself is not interpretable, so only its values (which cross-check exactly against Table 1) were relied on.
- The missing hyperparameter values (N, n, K, tau_cr, K_rft, climb count) are absent from the paper text itself, not lost in extraction; the provided text ends cleanly at the Figure 2 caption with no cut-off.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.