Ask Before You Optimize: Dynamic Pre-Formulation Clarification for Interactive Optimization
Sihan Ge, Yichen Lin, Chenyu Zhou, Jianghao Lin, Tao Yao, Dongdong Ge
math.OC · cs.AI
Quick verdict
OR-Clarify + InterOPT: benchmark and method for pre-modeling clarification in OR; strong Choice-setting gains, heavy interaction cost, no significance testing.
Summary
The paper targets a gap in LLM-based optimization modeling: agents formulate models from incomplete business briefs without knowing when to ask for clarification. The authors introduce OR-Clarify, a benchmark of 100 cases (178 hidden slots; 75 P0, 83 P1, 20 P2) built by masking formulation-critical facts from complete OR task records, with a fact-bounded simulated user, a post-hoc LLM judge, and metrics for exact slot recovery, silent assumptions, stopping quality, and interaction cost. They also propose InterOPT, a two-stage framework in which Stage 1 maintains a persistent ledger of unresolved formulation gaps and Stage 2 generates gap-anchored candidate questions and decides ask-versus-stop. In choice-based experiments (DeepSeek V4 Pro, 100 cases x 5 runs), InterOPT reaches 0.675 Core Exact versus 0.517 for ReadyGate and 0.506 for MC-D, but at 10.7 turns and 10.0 questions per run versus roughly 3.3 for MC-D. In the open-ended setting, InterOPT (0.538 Core Exact) is outperformed by the adapted ORPilot (0.583) and GATE (0.560). Ablations attribute recovery gains mainly to gap discovery and interaction efficiency to action selection. Main limitations: no statistical significance tests, endpoint comparisons that conflate a 3-4x question budget, unquantified judge reliability, and method comparisons using a single tested model.
Review complete
Assessment at a glance
5
risk flags identified
● high 0
● medium 3
● low 2
5 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- OR-Clarify is the first OR-specific benchmark to jointly evaluate formulation-gap recovery, readiness decisions, and interaction cost before LLM autoformulation.
- In the choice-based protocol, InterOPT substantially outperforms all evaluated training-free baselines in exact slot recovery (Core Exact 0.675 vs 0.517 for ReadyGate and 0.506 for MC-D).
- In the open-ended protocol, InterOPT remains competitive but does not dominate (0.538 Core Exact vs 0.583 for the ORPilot adapter).
- All off-the-shelf LLMs tested stay below 60% Core Exact with substantial silent assumptions, so pre-formulation clarification remains unsolved.
- Ablations attribute recovery gains mainly to Stage 1 gap discovery and interaction-efficiency gains to Stage 2 guided action selection.
Strengths
Novel, formally grounded task definition
Section 3 defines formulation (in)completeness via whether alternative plausible completions change the induced formulation (Eq. 3.1-3.2), reframing OR assistance as a 'selective completeness decision' rather than one-shot modeling.
Leakage-controlled evaluation harness
Section 4.1-4.2: masking is seeded and frozen before evaluation, severity labels are assigned after masking so they cannot bias fact selection, the simulated user answers only from private facts, the judge scores post-hoc against frozen rubrics, and the protocol detector is passive and supplies no recovery information.
Unusually candid negative results
Diagnostic depth beyond headline metrics
Reproducibility scaffolding
Weaknesses
No inferential statistics
Budget conflation in headline comparison
Judge/simulator validity unquantified
Open-setting baselines are adapters
Single tested model for all method comparisons
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
The paper defines **pre-formulation clarification** as a standalone task: before an LLM agent turns a business brief into an optimization model, it must decide whether the specification is model-ready and recover formulation-critical missing facts (Section 3, Eq. 3.1-3.2 define formulation (in)completeness via whether alternative plausible completions change the induced formulation structure). Two artifacts follow. (1) **OR-Clarify**: 100 cases / 178 hidden slots (75 P0, 83 P1, 20 P2; mean 1.78 slots per case), produced by decomposing complete OR task records, screening maskable facts, seeded masking of roughly half of eligible facts, and annotation with evidence, simulated answers, recovery rubrics, and severity (Section 4.1). (2) **InterOPT**: Stage 1 maintains a persistent gap ledger (at most three new gaps per turn across six gap categories); Stage 2 generates three gap-anchored candidate questions, a selector picks one, and the ask-versus-READY_TO_MODEL decision stays with the model (Section 5, Algorithm 1). Headline claims: in the Choice protocol InterOPT substantially beats all evaluated training-free baselines on exact recovery; in the open-ended setting it is competitive but not dominant; all off-the-shelf LLMs remain below 60% Core Exact (Table 1, best: Opus-4.8 at 0.583).
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 5 strengths & 5 weaknesses, analyzed in detail
- 5 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What is the OR-Clarify benchmark?
OR-Clarify is a benchmark of 100 operations research problem cases in which formulation-critical facts (178 hidden slots total) are deliberately withheld from the public problem brief. LLM agents must interact with a simulated user to recover the missing facts and decide when they have enough information to build an optimization model; scoring measures exact fact recovery, silent assumptions, stopping quality, and interaction cost.
How much does InterOPT improve over baselines?
In choice-based experiments (DeepSeek V4 Pro, 100 cases x 5 runs), InterOPT reaches 0.675 Core Exact recovery versus 0.517 for ReadyGate and 0.506 for MC-D, but uses about 3.3x more turns and 4.3x more questions than MC-D. In free-form questioning it scores 0.538, below the adapted ORPilot (0.583) and GATE (0.560).
Is code and data available for this paper?
Yes. The paper links code at github.com/AIOR-Research/InterOpt and the dataset at huggingface.co/datasets/AIOR-Research/OR-Clarify; masking is seeded and deterministic, and evaluation temperatures are fixed (agent 0.2, simulator/judge 0.0).
What are the main limitations of this work?
InterOPT's gains cost far more interaction than baselines (about 10.7 turns vs 3.3 in the choice setting); stopping remains imperfect (40.6% of open-setting runs were flagged premature); silent assumptions are reduced but not eliminated; and the paper reports no statistical significance tests, does not validate its LLM judge against human raters, and runs all method comparisons with a single tested model.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Figures 1-4 appear only as captions in the extracted text; Figure 4's curve behavior (where baselines plateau) is taken from the authors' Section 6.4 prose, not from the plots themselves.
- The extraction contains the main text through reference [28] and no appendix; any supplementary material beyond the main text was not part of the provided content and could not be reviewed.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.