SAILOR: Solver-Assisted Interactive LLM-based Optimization Recovery
Shaghayegh Sadeghi, Stephen L. Smith, David C. Del Rey Fern'andez
cs.AI
Quick verdict
PoC system asks users for numbers missing from LLM-translated optimization models; promising, but evidence rests on idealized feedback and circular masking.
Summary
The paper presents SAILOR, a proof-of-concept pipeline for recovering numerical information that users omit from natural-language optimization descriptions. An LLM first translates the description into Gurobi code while recording every value it had to guess; AST parsing separates known from guessed literals; Gaussian beliefs and solver-perturbation importance scores then prioritize clarification questions, whose answers trigger belief updates, AST rewrites, and re-solves. Benchmarks are built by masking decision-critical constants from seven established datasets (1,723 of 1,732 instances evaluated). An idealized simulator answers with ground-truth values, revealing the exact number after every question type. Exact objective-value agreement ranges from 27.0% (ComplexLP) to 87.6% (NL4LP), versus 5.6-34.2% for a no-interaction baseline, using 1.4-5.7 questions per instance. Ablations show solver-derived importance scoring yields no clear aggregate benefit (66.4% vs 66.1% macro-average) and the released default disables it; swapping the translator for GPT-4o-mini costs 27.8-54.8 points, identifying translation fidelity as the binding constraint. Key limitations, largely self-flagged: circular masking that matches the method's own importance signals, idealized rather than human feedback, no significance testing, and confounded baseline comparisons.
Review complete
Assessment at a glance
4
risk flags identified
● high 2
● medium 2
● low 0
5 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- Asking targeted clarification questions about guessed literals raises exact objective-value agreement from 5.6-34.2% (no interaction) to 27.0-87.6% across seven masked benchmarks under an idealized ground-truth simulator.
- AST-based extraction of guessed constants plus solver perturbation yields a reusable benchmark-construction procedure that retains ground truth for automated scoring.
- Translation fidelity, not question prioritization, is the binding constraint: swapping GPT-5.3-Codex for GPT-4o-mini costs 27.8-54.8 percentage points of exact agreement.
- Solver-derived importance scoring provides no clear aggregate benefit (66.4% vs 66.1% macro-average), so the released default disables it.
Strengths
Novel, well-motivated problem setting
Interactive completion of optimization descriptions with missing numbers is excluded from existing NL-to-solver benchmarks by construction (Sec 1-2); Sec 3 formalizes recovery via ErrObj on objective values and states precisely what the metric does and does not establish.
Reusable benchmark-construction recipe
Sec 5 and Appendix A give a complete, transferable procedure: AST constant extraction with structural roles, x{0.5,0.8,1.2,2.0} perturbation ranking, importance-driven masking with natural phrasings, and retained ground truth for automated scoring.
Exceptional self-critical reporting
Reproduction-grade specification
Locating the binding constraint
Weaknesses
Circular benchmark construction
Idealized feedback leaves the core mechanism untested
No statistical inference
Confounded headline comparison
Belief model invalid for common parameter types
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
SAILOR targets a gap the authors argue existing natural-language-to-optimization benchmarks exclude by construction: descriptions that omit the numbers a solver needs. The pipeline (i) translates the description to Gurobi code while recording every guessed literal, (ii) separates known from guessed constants via AST parsing plus regex matching against the input (Sec 4.3), (iii) maintains Gaussian beliefs and solver-derived importance scores, (iv) asks scored clarification questions, and (v) rewrites literals through AST edits and re-solves (Sec 4.7). Claims are carefully calibrated: the abstract and Sec 7 frame this as a proof-of-concept under controlled branch-and-reveal feedback that does not measure human interaction or structural model repair. The headline empirical claim is that interaction raises exact objective-value agreement from 5.6-34.2% (no-interaction) to 27.0-87.6% across seven masked benchmarks (Table 2), at 1.4-5.7 questions per instance. A secondary, unusually candid claim is that the most heavily engineered mechanism - solver-derived question prioritization - shows no clear aggregate benefit.
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 5 strengths & 5 weaknesses, analyzed in detail
- 4 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What is the SAILOR system?
SAILOR is a proof-of-concept pipeline for optimization problems described in natural language: an LLM translates the description into Gurobi code, flags every number it had to guess, asks the user targeted clarifying questions about the most consequential ones, rewrites the model program, and re-solves it. The paper evaluates it on 1,723 instances drawn from seven public benchmarks.
How accurate is SAILOR?
Under an idealized simulator that always reveals exact values, the share of instances whose optimal objective value exactly matches the benchmark answer ranges from 27.0% (ComplexLP) to 87.6% (NL4LP), versus 5.6-34.2% if the translator's initial guesses are simply kept. The system asks 1.4-5.7 questions per instance on average.
Was SAILOR tested with real human users?
No. All evaluation uses an automated simulator that answers with ground-truth benchmark values; the authors explicitly state the study does not measure performance with human users, whose real answers could be approximate, incomplete, or inconsistent.
Does the solver-based question prioritization help?
According to the paper's own ablation, no clear aggregate benefit was found: macro-average exact agreement was 66.4% with solver-derived importance scoring versus 66.1% without it, so the released code disables solver-derived scoring by default.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Figures 1-5 are available only as text annotations in the extraction; visual details such as axis scales and distributions could not be verified.
- Review is based solely on the provided extraction of arXiv:2609.13945v1; the GitHub repository and benchmark release were not externally verified.
- Extraction appears complete through Appendices A-B, but any supplementary material beyond it was not visible.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.