Controllable Image Captioning with Prompt-Conditioned Scene Rewards
Jongyeop Hyun, Taeyoung Kim, Hyounghun Kim
cs.CV · cs.CL · cs.LG
Quick verdict
FoCUS: GRPO with signed scene-graph rewards for controllable captioning; solid ablations, but judge circularity and caption-quality caveats remain.
Summary
The paper proposes FoCUS, a controllable image captioning method that steers vision-language models toward user-specified semantic emphases (general, attribute, relation, foreground, background) through natural-language prompts. Generated captions are parsed with spaCy, object mentions are matched to ground-truth scene-graph objects via Sentence-BERT mutual-best matching with a validity threshold (τ=0.5), and attribute/relation alignment is scored 0–5 by a Qwen3-30B-A3B judge with chain-of-thought verification. Component scores are aggregated with prompt-specific signed weights (e.g., −1.0 on off-scope components) and optimized with GRPO after SFT. The authors also introduce SCoPE, a 189-image benchmark with contrastive Include/Avoid atomic-fact lists built from Gemini-3-Flash captions, scoring Coverage, Adherence, Faithfulness and their harmonic mean. On Qwen2.5-VL-3B and InternVL3-2B, FoCUS raises SCoPE overall from 15.66→31.74 and 25.63→36.86 over zero-shot, beating CLIP-reward and CompreCap-reward GRPO baselines; ablations attribute the largest gains to CoT verification and the stronger judge (+7.60 overall on InternVL3-2B). Human MTurk correlation (ρ=0.5704) far exceeds a CompreCap reweighting (0.2291). Limitations: reliance on proprietary models (GPT-5 SFT data, Gemini benchmark construction with the generator as its own verifier), small training sets (280 GRPO images), LLM-judge dependence, and general-caption metrics that slightly regress on InternVL3 (CIDEr 0.111→0.091), while the reported zero-shot CIDEr of 0.004 for Qwen2.5-VL is anomalously low.
Review complete
Assessment at a glance
4
risk flags identified
● high 0
● medium 3
● low 1
5 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- A prompt-conditioned control objective with signed scene-graph weights, optimized via SFT+GRPO, improves controllable captioning across all four emphasis categories on two VLM backbones.
- Stricter object matching (mutual-best, τ=0.5), CoT-based attribute/relation verification, and a stronger judge (Qwen3-30B-A3B) jointly account for +7.60 SCOPE Overall (29.26→36.86) on InternVL3-2B (Table 3).
- SCoPE's contrastive Include/Avoid design measures controllability and aligns with human MTurk judgments (Spearman ρ=0.5704) far better than a CompreCap-based reweighting (ρ=0.2291).
- FoCUS improves controllability while largely maintaining general caption quality, though CIDEr and ROUGE-L slightly regress on InternVL3-2B (Table 13).
Strengths
The core mechanism is isolated by a decisive ablation
Appendix J.2 shows Fixed Reward Weights (23.17) and Simply Up-weighted (22.35) SCOPE Overall fall far below FoCUS (31.74) on Qwen2.5-VL-3B, directly validating the signed prompt-conditioned weighting rather than scene-graph rewards in general.
Reward-noise engineering is quantified, not asserted
Table 3 decomposes the reward-hardening contributions on InternVL3-2B: validity threshold +0.53, CoT verification +4.79, judge swap +5.06, all combined +7.60 (29.26→36.86), exactly matching the paper's noise-reduction narrative in Sec. 3.1 and Appendix A.
Benchmark is calibrated against humans and across judges
Statistical hygiene above field norm
Honest contextualization
Weaknesses
Abstract overstates general-caption preservation
Generator–verifier circularity in SCoPE construction
Headline gains are anchored to weak zero-shot baselines
Small, model-generated training data
Hand-set negative weights with fragile extremes
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
FoCUS (arXiv 2609.00709; POSTECH; ACL-formatted with an Action Editor acknowledgment) targets controllable image captioning: given a natural-language emphasis prompt (General/Attribute/Relation/Foreground/Background), a vision-language model should cover requested semantics and suppress out-of-scope content. Three claims: (1) a prompt-conditioned signed scene-graph reward, trained via SFT+GRPO, improves controllability; (2) stricter object matching, CoT-based attribute/relation judging, and a stronger judge make the reward reliable enough for RL; (3) the new SCoPE benchmark measures contrastive controllability and aligns with human preferences far better than CompreCap reweighting. Scope is appropriately bounded to 2–3B backbones; the authors do not claim parity with frontier commercial models (Table 16 shows Gemini 2.5 Pro at SCOPE Overall 55.86, above their tuned models).
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 5 strengths & 5 weaknesses, analyzed in detail
- 4 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What is FoCUS in controllable image captioning?
FoCUS (Fine-grained Captioning Control Using Scene Rewards) is a training method that lets users steer an image captioning model with natural-language prompts, e.g. 'focus on attributes' or 'focus on the background'. It parses generated captions into scene-graph components (objects, attributes, relations), scores them with embedding matching plus an LLM judge, combines the scores with signed prompt-specific weights (positive for requested content, negative for off-scope content), and optimizes the resulting reward with GRPO reinforcement learning on top of supervised fine-tuning.
What is the SCoPE benchmark and how does it score captions?
SCoPE (Semantic Control and Precision Evaluation) is a 189-image benchmark for controllable captioning. For each image and control category it holds contrastive Include and Avoid lists of atomic facts. A generated caption is scored on Coverage (fraction of Include facts present), Adherence (1 minus the fraction of Avoid facts leaked), and Faithfulness (share of matched facts not contradicted), combined into an overall score by harmonic mean. It correlates with human MTurk judgments (Spearman ρ≈0.57) much better than a CompreCap-based reweighting (ρ≈0.23).
Does FoCUS hurt normal (general) captioning quality?
Mostly no, with caveats. On 5,000 DOCCI test images the paper reports largely maintained general-caption quality: CAPTURE and general-prompt CompreCap improve for both backbones. However, CIDEr drops from 0.111 to 0.091 and ROUGE-L from 0.220 to 0.212 on InternVL3-2B, and the reported Qwen2.5-VL zero-shot CIDEr of 0.004 is anomalously low, making that comparison hard to interpret.
Is the FoCUS code and SCoPE dataset available?
The paper states that code and dataset are publicly available via focus-emnlp2026.github.io. Training uses open-weight backbones (Qwen2.5-VL-3B, InternVL3-2B) and open-weight judge models (Qwen3 family), but SFT captions were generated with GPT-5 and SCoPE facts with Gemini-3-Flash, so exactly reproducing the data-construction pipelines requires access to those proprietary APIs.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Figure-embedded numerals (Figs. 2, 3, 5, 8) are scrambled in the extraction; the figure-level values cited here (e.g., 167.57/110.38 tokens, penalty −3.0) are taken from the surrounding prose and captions, which quote them explicitly.
- No network access in this review, so the claimed public code/dataset URL (focus-emnlp2026.github.io, footnote 1) could not be verified.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.