PolyBridgeBench: Benchmarking Multimodal LLMs for Physics-Grounded Bridge Design
Zicheng Zhao, Dongyin Chen, Rui Xu, Yinghui Xu
cs.AI
Quick verdict
Executable MLLM bridge-design benchmark that separates legality, function, and repair over 189 levels; sound design, but single-run results lack variance.
Summary
PolyBridgeBench is an executable benchmark asking whether multimodal LLMs can synthesize complete node-member-material bridge topologies that pass deterministic legality checks and then function under native Poly Bridge physics. Across 189 levels (149 Core Structural, 40 Compositional), models receive an editor image plus structured constraints and get up to three LLM calls and three physics attempts, with failed rollouts returning four-frame temporal contact sheets for repair; Valid@k, Success@k, and Recovery@k are reported separately. Deterministic validity proves a poor proxy for function: under loose budgets Qwen3-VL-Plus reaches 92.1% Valid@3 but only 12.2% Success@3. Budget enforcement reorders rankings: Gemini-3-Flash-Preview leads loose-budget Success@3 at 84.7%, while Claude-Sonnet-5 leads strict-budget at 50.8% versus Gemini's 24.9%. Compositional levels are hardest — under strict budgets only Gemini (5.0%) and Claude (15.0%) score non-zero — and the best Recovery@3 is Claude's 22.0%. Error decomposition localizes model-specific bottlenecks: Claude fails mostly after validation (93-94% RUN_FALSE_AFTER_VALID), whereas GPT-5.2 and Kimi-K2.5 are dominated by EDGE_TOO_LONG validator errors. Key limitations: one run per model-level-condition with no confidence intervals or significance tests; single-pixel success detection; the proprietary game is not redistributed; and the post-failure feedback package is never ablated.
Review complete
Assessment at a glance
6
risk flags identified
● high 0
● medium 2
● low 4
5 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- Deterministic legality is a poor proxy for dynamic functionality: e.g., Qwen3-VL-Plus attains 92.1% Valid@3 but only 12.2% Success@3 under loose budgets.
- Strict per-level material budgets reorder model rankings: Claude-Sonnet-5 leads strict Success@3 at 50.8% while Gemini-3-Flash-Preview leads loose at 84.7% (Gemini strict: 24.9%).
- Compositional constraint combinations are the hardest condition: under strict budgets only Gemini (5.0%) and Claude-Sonnet-5 (15.0%) achieve non-zero Success@3 on the 40 Level-2 levels.
- Post-failure recovery under a fixed 3-call/3-execution budget is limited, with the best strict Recovery@3 at 22.0% (Claude, 20/91 eligible episodes).
- Failure bottlenecks are model-specific and separable into constraint-encoding errors (EDGE_TOO_LONG, OVER_BUDGET) versus post-validation dynamic failures (RUN_FALSE_AFTER_VALID).
Strengths
Layered metric protocol with honest conditional denominators
Valid@k (Eq. 2), Success@k (Eq. 3), and Recovery@k (Eq. 4) are defined separately; Appendix D.2 gives an explicit accounting table for edge cases (validator rejections vs physical failures) and frames Recovery@3 as a conditional diagnostic, not an aggregate success rate — this localizes the stage of failure rather than collapsing it into one number.
Execution-based, one-to-many correctness with a well-specified validator gate
Any legal topology that completes vehicle traversal counts (no reference matching). Table 7 enumerates nine deterministic checks (anchor integrity via 10^-3 tolerance, member lengths, duplicate edges, union-find anchor connectivity, clearance polygons, cost); invalid drafts consume a call but no physics attempt, cleanly separating rule errors from dynamic failures.
Unusually strong auditability and provenance
Diagnostic error decomposition that separates model regimes
Candid threats-to-validity disclosure
Weaknesses
No statistical rigor behind headline rankings
Fragile, opaque success detection
The repair-evidence claim is never causally tested
Cross-model comparability confounds in request configuration
Minor internal numeric inconsistency
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
PolyBridgeBench frames multimodal bridge construction as complete-topology generation under an executable protocol: given an editor image, structured constraints, and native anchors, an MLLM must emit a full node-member-material graph that (i) passes deterministic legality checks and (ii) lets all vehicles traverse under Poly Bridge's native dynamics. 189 levels from 27 procedural families split into a 149-level Core Structural Suite and a 40-level Compositional Challenge Suite (Fig. 2, Tables 2-4). Six models are evaluated (Gemini-3-Flash-Preview, Qwen3-VL-Plus, Gemma-4-31B-IT, Kimi-K2.5, GPT-5.2, Claude-Sonnet-5) under loose ($50k uniform) and strict (per-level calibrated, $6.5k-$42k) material budgets, with at most 3 LLM calls and 3 physics attempts per episode. Central claims: legality is a poor proxy for function; strict budgets reorder rankings; post-failure recovery is limited (best strict Recovery@3 = 22.0%); failures are model-specific and localizable via the layered metrics.
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 5 strengths & 5 weaknesses, analyzed in detail
- 6 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What is PolyBridgeBench?
An executable benchmark that asks multimodal LLMs to design bridges for the physics game Poly Bridge. Each model gets a scene image plus structured constraints and must output a complete node-member-material structure that passes rule checks and lets vehicles cross under the game's real physics, with up to three design-and-repair rounds per level across 189 levels.
Which AI model performed best on PolyBridgeBench?
It depends on the setting. With loose material budgets, Gemini-3-Flash-Preview scored highest (84.7% final success across all 189 levels). With strict per-level budgets, Claude-Sonnet-5 was best (50.8%), ahead of Gemini (24.9%); all other models scored below 15%. Under strict budgets only Gemini and Claude scored above zero on the hardest compositional levels.
Why do rule-valid bridge designs still fail in the benchmark?
The deterministic checks only verify rules such as member length, connectivity, materials, and cost — not whether the structure carries load. For example, Qwen3-VL-Plus produced rule-legal designs in 92.1% of episodes under loose budgets, but only 12.2% of its designs actually let vehicles cross under the physics simulation.
Is the PolyBridgeBench benchmark reproducible?
Partially. The authors describe released level generators, controller code, full run trajectories, and integrity manifests, and all metrics can be recomputed from the logged trajectories. However, native execution requires a legally obtained copy of Poly Bridge plus screen automation with fixed-resolution pixel-threshold success detection, so exact replication is sensitive to the local environment and game version.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Figures 1-5 are not visually analyzable from the extracted text; all figure-derived numbers cited here come from the prose, which quotes the key percentages.
- Table 1's layout is garbled in extraction; values were reconstructed by column alignment and cross-checked against Appendix E (Table 13 recovery counts match Table 1 Recovery@3).
- Model identifiers (e.g., 'Claude-Sonnet-5', 'GPT-5.2') are provider-mutable aliases documented only via Table 11 snapshots; findings bind to those access-time snapshots, not to stable model versions.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.