OmniHarness: Harnessing Generalizable Visual Generation via Symbolic Policy Learning
Xu Xu, Jinxiu Liu, Zhangbo Qiao, Jiaxing Lu, Xiangyu Zhang, Yubin Gu, Fangwei Ning, Yan Shi
cs.LG · cs.AI
Quick verdict
声称符号策略学习（symbolic-policy learning）提升了视觉生成智能体（ComfyBench 上 Creative 分辨率达 95.0%）；证据充分，但数值一致性需要核实。
Summary
OmniHarness 是一个免训练框架，多模态 LLM 智能体在其中通过积累符号策略而非微调权重来提升视觉生成能力。自主探索循环自动提出接近智能体预估能力边界（能力新颖度 × Goldilocks 可学性分数）的实践任务，通过 ComfyUI 中的规划-编写-验证-修复流水线执行这些任务，并将验证过的执行提炼为可复用的工作流模板以及失败库；该库的冻结快照可以插入到外部智能体中。在 ComfyBench 上，Codex GPT-4o 配置达到了 100% Pass 和 92.5% Total Resolve，其中 Creative 任务为 95.0%，而最强基线 SymbOmni 为 67.5%。论文进一步报告了 GenEval 0.997、WISE WiScore 0.86 以及 KRIS-Bench 77.33。消融实验将收益主要归因于自主探索（若没有它，Creative 任务会下降 22.5 分）、任务选择分数和中间验证；相同的快照提升了所有三个外部智能体框架，且仅基于图像的策略可迁移到未见过的视频任务中（总体为 85.9%）。主要局限性：没有重复运行或显著性检验；子集/总度量的内部算术不一致（例如，Creative Total 95.0 超过了两个子集）；内部验证与基准评分器之间可能存在对齐；以及跨模态比较中骨干网络的混合。代码和完整提示词已发布。
Review complete
Assessment at a glance
5
risk flags identified
● high 1
● medium 3
● low 1
5 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- Distilling verified executions into symbolic policies (workflow templates with preconditions plus a failure library) transfers across a visual generation task family where storing instance-specific solutions does not, e.g. 95.0% vs 67.5% Creative Resolve on ComfyBench (Table 1).
- Self-directed inquiry acquires useful policies before downstream tasks arrive: removing it drops ComfyBench Total Resolve 92.5%→87.0% and Creative 95.0%→72.5% (Table 6).
- Planning, intermediate verification, and localized recovery during execution limit error propagation: removing them drops Complex Resolve from 83.3% to 55.0%, 68.3%, and 76.7% respectively (Table 8).
- A frozen inquiry-learned policy snapshot improves three external visual agent frameworks plug-and-play (Table 16) and transfers to unseen video generation after image-only practice, reaching 85.9% Total Resolve (Table 17).
Strengths
Broad, well-controlled evaluation
Six benchmarks (ComfyBench, GenEval, GenEval2, WISE, Reason-Edit, KRIS-Bench); same-backbone framework comparisons in Table 13 isolate harness from model effects (GPT-4o: OmniHarness 89.5 vs ComfyMind 83.0 vs ComfyAgent 32.5 Total Resolve), plus backbone and planner ablations (GPT-5.5, Codex GPT-5.5 up to 95.0).
Granular, high-magnitude ablations
Inquiry removal drops Creative Resolve 95.0→72.5 (Table 6); task-selection scores isolated in Table 7; planning/intermediate-verification/localized-recovery removal drops Complex Resolve to 55.0/68.3/76.7 (Table 8); iteration curve saturates at N=50 (A.4.1: 92.5→93.5 going to 100).
Explicit leakage controls
Transfer evidence beyond the home framework
Implementation transparency and honest reporting
Weaknesses
Arithmetically inconsistent subset/total reporting
No uncertainty quantification
Possible verifier–grader circularity
Comparison asymmetries
Inquiry uses benchmark source images
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
OmniHarness (arXiv 2609.16057) is a method paper on training-free self-improvement of visual-generation agents. The authors claim three mechanisms: (1) symbolic policy learning — distilling verified ComfyUI executions into reusable workflow templates with preconditions, plus a failure library (§3.3, Tables 10–11); (2) feedback-guided execution — plan → Code-as-Policy write → reversible compile → four-level verification → localized repair (§3.2, A.1.3); (3) self-directed inquiry — autonomously generated practice tasks selected by capability novelty × a Goldilocks competence-frontier score C(τ)=4r̄(1−r̄) over Wilson-lower-bound reliability (§3.1, A.1.2). Headline claims: 92.5% Total Resolve on ComfyBench (Codex GPT-4o, Table 1), 95.0% Creative Resolve (+27.5 pp over SymbOmni), GenEval 0.997, plug-and-play gains for three external agents, and image-to-video policy transfer.
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 5 strengths & 5 weaknesses, analyzed in detail
- 5 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What is OmniHarness?
A training-free framework in which a multimodal LLM agent improves at visual generation by learning a library of reusable symbolic workflow policies — ComfyUI workflow templates with preconditions plus a library of recorded failures and fixes. It practices on self-generated tasks, verifies intermediate outputs during execution, and keeps all model weights frozen.
What are OmniHarness's main results?
On ComfyBench it reports 100% Pass and 92.5% Total Resolve (Codex GPT-4o configuration), including 95.0% on Creative tasks versus 67.5% for the strongest prior baseline SymbOmni. It also reports GenEval 0.997, WISE WiScore 0.86, KRIS-Bench 77.33, and the best Understanding-scenario editing metrics on Reason-Edit.
Does OmniHarness fine-tune any model?
No. All learning happens in the symbolic policy and failure libraries via execution feedback; model parameters stay frozen, and the learned library is exported as a frozen snapshot that other agents can reuse without any training.
Can other agent systems reuse OmniHarness's learned policies?
Per the paper, inserting the same frozen snapshot without fine-tuning raises Total Resolve on ComfyBench for ComfyAgent (32.5%→57.0%), ComfyMind (83.0%→88.0%), and SymbOmni (86.0%→89.0%), and policies learned from images only still solve unseen video tasks at 85.9% overall.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Figures 1–22 are not analyzable from the extracted text; quantitative claims about curves (e.g., Figure 8 iteration numbers, library growth in Figures 9–11) are taken from captions and prose only.
- Part of Table 3 (GenEval2) and surrounding Section 4.2 text were truncated in extraction; GenEval2 baseline rows are only partially visible.
- Per-subset task counts (T2I vs I2I splits) are not stated in the text, which limits full verification of the subset/total arithmetic flagged in this review.
- Review based on the extracted v1 text (13 Sep 2026); appendix sections A.4.3–A.9 are partially truncated.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.