An Evolutionary Agentic Approach for Open-ended Image Quality Perception
Zhenchen Tang, Bo Peng, Zichuan Wang, Songlin Yang, Leilei Cao, Fengjie Zhu, Jing Dong
cs.CV · cs.AI
Quick verdict
Agent-built evaluation protocols cut MLLM holistic bias in IQA (HOR 44.4→8.6%); promising, but central eval is a 180-image self-built set without stat tests.
Summary
The paper identifies 'holistic bias' in MLLM-based image quality assessment: when scoring unseen perceptual dimensions, models reuse generic quality priors and inflate scores (Q-Align reaches a 58.8% Holistic Override Rate on the Tactile Satisfaction and Human Vitality subset). It proposes PACE, a training-free multi-agent framework that treats open-ended IQA as protocol construction: a Planning Agent decomposes a target dimension into 3-5 observable sub-dimensions, Visualizer and Critic agents iteratively refine verifiable VQA probes, and a Generator synthesizes four candidate images for one-time human calibration of two scoring anchors. Dual-track scoring fuses protocol-based absolute perception (softmax-weighted logit expectations) with anchor-relative positioning, linearly projected into the anchor score range. Across four tiers - traditional IQA (KonIQ, SPAQ, KADID, etc.), structural integrity (HandEval, PIPAL), context-aware aesthetics (TAD66k), and six self-constructed open-ended dimensions (180 images) - PACE improves its Qwen2.5-VL backbone (e.g., KADID PLCC 0.576 to 0.782; Tier-4 overall PLCC 0.493 to 0.714) and reduces average HOR from 44.4% to 8.6%. An mPLUG-Owl2 variant confirms backbone generality. Main limitations: the centerpiece Tier-4 benchmark is small and author-built from Gemini prompts and Bagel-generated images; no confidence intervals or significance tests are reported; fast-thinking Tier-1 results reuse supervised expert models; and protocols are generated with the test image as visual grounding, complicating the claimed reusability.
Review complete
Assessment at a glance
5
risk flags identified
● high 0
● medium 4
● low 1
5 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- MLLM-based IQA models exhibit systematic holistic bias on unseen dimensions, quantified by a Holistic Override Rate averaging 44.4% for Q-Align and reaching 58.8% on the Tactile Satisfaction and Human Vitality subset (20/34 GT-low images).
- PACE, a training-free multi-agent framework that constructs verifiable VQA-probe protocols, reduces average HOR from 44.4% to 8.6% while improving PLCC/SRCC with human MOS across four evaluation tiers.
- Protocol-based evaluation lifts the Qwen2.5-VL backbone substantially (KADID PLCC 0.576 to 0.782; Tier-4 overall PLCC 0.493 to 0.714) and generalizes to mPLUG-Owl2 (+0.143 KonIQ PLCC).
- Two human-scored calibration anchors, obtained from only four synthesized candidate images per dimension, suffice to align model perception with human score scales via dual-track fusion.
Strengths
Formalizes a real failure mode with a measurable metric
Sec. 3 defines Holistic Override Rate (HOR, Eq. 1) with explicit thresholds (MOS<=3.0 vs predicted>=4.0); Appendix A.3 gives exact counts (36/81 images for Q-Align) and Table 8 shows three independent holistic scorers (Q-Align 3.75, Q-Scorer 3.06, DeQA-Score 3.59 mean) all inflate the same 81 GT-low images vs human GT 2.09.
Genuine training-free gains over the backbone
Ours (Slow) lifts Qwen2.5-VL from 0.576 to 0.782 PLCC on KADID (Tab. 1) and 0.493 to 0.714 PLCC on Tier-4 overall (Tab. 4); mPLUG-Owl2 experiments (Tab. 9) add +0.143 KonIQ PLCC, supporting backbone generality beyond one model.
Extremely cheap per-dimension calibration
Annotation hygiene on the new benchmark
Honest cost reporting and component ablations
Weaknesses
Centerpiece benchmark is tiny, synthetic, and author-built
No uncertainty quantification anywhere
Training-free framing blurs expert reuse on Tier 1
Protocol generation conditioned on the test image
Track 2 contribution partly confounded by range rescaling
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
PACE (arXiv 2609.22942) targets open-ended image quality assessment (IQA): scoring an arbitrary, user-specified perceptual dimension (e.g., tactile satisfaction) for which no training data exists. The central diagnosis is **holistic bias** - MLLMs reuse generic quality priors on unseen dimensions - formalized in Sec. 3 via the Holistic Override Rate (HOR; Eq. 1, human MOS <= 3.0 vs predicted >= 4.0). The claims: (1) existing methods reach HOR up to 58.8% (Q-Align, 20/34 GT-low images on the Tactile Satisfaction + Human Vitality subset, A.3); (2) PACE, a training-free multi-agent framework, cuts average HOR from 44.4% to 8.6% while improving correlations across four evaluation tiers; (3) two human-annotated anchors per dimension suffice for scale alignment. The reframing of evaluation as on-the-fly protocol construction is the paper's most interesting conceptual contribution.
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 5 strengths & 5 weaknesses, analyzed in detail
- 5 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What is holistic bias in image quality assessment?
In this paper, holistic bias is the tendency of multimodal LLMs, when asked to score an image on an unfamiliar quality dimension (e.g., tactile satisfaction), to fall back on their generic impression of overall image quality - so a beautiful image with broken text rendering still receives a high text-fidelity score. The authors quantify it with the Holistic Override Rate (HOR): the share of images humans rate low (MOS <= 3.0) that the model scores high (>= 4.0), measured at up to 58.8% for the Q-Align baseline.
How much human annotation does PACE need for a new quality dimension?
Very little: a Generator agent synthesizes four candidate images from the constructed evaluation protocol, humans score them, and the best and worst become two calibration anchors. During benchmark evaluation, anchors are instead taken from the training split's highest- and lowest-MOS images, and no model parameters or protocols are updated.
Is PACE training-free, and how fast is it?
PACE requires no training, but the first image on a new dimension is slow: protocol construction averages 22.72 s/image on an A100 versus 0.12 s for direct Qwen2.5-VL scoring. Once a protocol is cached, the fast-thinking path drops to 0.67 s/image - a reported 33.8x speedup over the slow path.
What are the main weaknesses of the PACE paper?
Its central open-ended benchmark is small and built by the authors themselves (180 images, 30 per dimension, all Bagel-generated from Gemini prompts); no statistical significance tests or confidence intervals are reported; the strong Tier-1 fast-mode numbers reuse supervised expert models rather than the training-free pipeline; and protocols are generated using the test image as visual grounding, which sits uneasily with the claimed cross-image reusability.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Figures 1-6 are available only as extracted captions/labels; the scatter-plot distributions behind HOR (Fig. 4/5) could not be visually inspected.
- Exact decoding parameters (temperature, sampling) for the agent pipeline are not present in the extracted text, limiting exact replication.
- Review based on arXiv v1 (19 Sep 2026); the release status of the Tier-4 dataset could not be verified from the text alone.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.