FPCO-Dialog: A Multi-Turn False-Premise Benchmark for Correction and Cooperation in Vision-Language Models
Jiayuan Ma, Yuqi Lu, Weiyang Guo, Chenrui Wang, Junyi Shu, Xuebo Liu, Min Zhang, Jing Li
cs.CL
Quick verdict
Controlled 20-VLM benchmark for repeated false-premise correction; rigorous validation, but headline class ordering is partly a lexical artifact.
Summary
FPCO-Dialog is a benchmark measuring how vision-language models respond when a user repeatedly describes an image through a false premise. The authors build 1,080 dialogues from MS COCO and Open Images, stratified across three visual-complexity levels, four object categories, and three false-premise classes (identity, attribute, location); each ten-turn dialogue opens with three premise-correct questions, then repeats the same minimally-edited false referring expression in turns 4–10. Twenty commercial and open-source VLMs are evaluated (216,000 responses), scored by two detectors (GPT-5.4 and Gemini-3.1-Pro-Preview) whose binary correction labels are averaged into CorrTP@K. Key results: correction tendency separates models persistently across turns; identity errors are corrected most (average CorrTP@10 ≈ 0.56), attribute errors intermediately (≈ 0.41), and location errors rarely (≈ 0.15); over-correction on clean turns is negligible (CorrFP ≈ 0.01). Validation is unusually thorough for an LLM-judge setup: detector agreement κ=0.942, human validation on 360 responses (κ 0.83–0.86), leave-one-judge-out self-bias checks, and bootstrap robustness analyses. Main limitations, largely self-reported: the identity–attribute ordering is not robust to lexical substitution choices (only 73.1% retention in the sensitivity study), the metric judges explicit repair rather than answer quality or pragmatic appropriateness, and annotation and curation were performed solely by the authors.
Review complete
Assessment at a glance
4
risk flags identified
● high 0
● medium 2
● low 2
4 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- A fixed repeated-false-premise protocol isolates correction-vs-cooperation behavior in VLMs in a way single-turn or uncontrolled multi-turn evaluations cannot.
- Cross-model differences in correction tendency are substantial and persist from turn K=4 through K=10.
- Correction varies systematically by false-premise class: identity ≈0.56, attribute ≈0.41, location ≈0.15 average CorrTP@10, with location frequently accommodated.
- Dual-detector scoring is reliable: detector κ=0.942, human–detector κ≥0.80 on 360 validated responses, and self-scoring bias does not change rankings.
- The class ordering is descriptive under the benchmark's substitution distribution — the identity–attribute gap is not robust to lexical variation.
Strengths
Controlled factorial design isolates a behavior other benchmarks conflate
A complete 3×4×3×30 stratification (complexity × category × premise class) with a fixed 10-turn schedule — three premise-correct turns then seven turns repeating the same minimally-edited false referring expression under a single-error constraint (§3.1–3.2) — cleanly separates correction behavior from task diversity.
Detector validation goes well beyond self-report
Dual-detector agreement 0.973 exact / κ 0.942 overall (Table 4); 360 stratified responses human-validated with human–detector κ 0.832–0.855 (Table 2, App. C); leave-one-judge-out check shows self-scoring bias immaterial (App. D: GPT-5.4 0.2939→0.2886, Gemini 0.5880→0.5906, ranks unchanged).
Specificity and decay are explicitly measured, not ignored
Honest calibration of its own headline finding
Weaknesses
Headline class ordering is partly a lexical artifact
No uncertainty quantification for model rankings
Author-only annotation end to end
Verbatim repetition may itself induce accommodation
Metric rewards explicitness, not correctness
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
FPCO-Dialog (arXiv 2609.03331; Harbin Institute of Technology, Shenzhen) addresses a narrow but real gap: how vision-language models behave when the *same* visually grounded false premise is repeated across dialogue turns. Where hallucination benchmarks test whether models generate unsupported content, here the inconsistency is user-supplied, and the measured behavior is whether the model explicitly repairs it (Correction) or answers the main question through it (Cooperation). The paper claims: (1) a controlled, stratified protocol isolates this behavior; (2) 20 VLMs differ substantially and persistently in correction tendency; (3) correction varies by false-premise class — identity ≈ 0.56 > attribute ≈ 0.41 > location ≈ 0.15 average CorrTP@10 — with the stated caveat that this ordering is descriptive under the benchmark's substitution distribution.
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 4 strengths & 5 weaknesses, analyzed in detail
- 4 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What is CorrTP@K in FPCO-Dialog?
It is the share of false-premise dialogue turns up to turn K where the evaluated vision-language model explicitly corrects or clearly repairs the user's injected false premise. Two detector models (GPT-5.4 and Gemini-3.1-Pro-Preview) judge each text response, their binary labels are averaged, and CorrTP@K aggregates them cumulatively from 0 (never corrects) to 1 (always corrects).
Which vision-language models correct false premises most and least in this benchmark?
Per Appendix E (Table 5, average CorrTP@10): Qwen-VL-Max leads at 0.688, followed by Gemini 2.5 Flash (0.654), Gemini 2.5 Pro (0.649) and Qwen3.6-Plus (0.633). Among frontier models GPT-5 is lowest at 0.079 (versus GPT-4o's 0.482), and small open models such as LLaVA-OneVision-7B are near zero (≈0.01).
Does 'cooperation' mean the model answered incorrectly?
No. The paper states explicitly (§4.4) that cooperation only means the model answered the user's main question without repairing the false premise; it is not an error label, and a correcting response may still answer the question, as the case studies in Table 1 show.
Is the FPCO-Dialog dataset publicly available?
Yes. The authors release dialogues, metadata, model outputs, detector labels, evaluation code and prompts at github.com/lab-klc/FPCO-Dialog under a mixed license: source images remain under their original COCO/Open Images licenses while author-generated components are released for research use.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Minor OCR artifacts exist in a few Appendix E–F table cells (merged or stray-quoted tokens, e.g., in LLaVA-OneVision-7B TurnCorr rows); all main-text numbers and the Table 2/4 statistics extract cleanly.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.