Lucida: Parse, Generate, and Place for Composable Real-to-Sim Scene Modeling
Minghan Qin, Yuang Wang, Xiuyu Yang, Yushi Long, Yujian Zhang, Ruihuan Wang, Kai Ye, Yangang Zhang, et al.
cs.CV · cs.AI
Quick verdict
ByteDance's Lucida adds a VLM gizmo-placement policy to parse-generate-place real-to-sim; strong on its own R2S benchmark, modest CA-1M gains, no code stated.
Summary
Lucida is a real-to-sim pipeline from ByteDance Seed that turns posed RGB-D video of cluttered indoor scenes into editable, individually placed 3D object assets. It keeps the parse-generate-place order but redistributes precision: parsing builds a scene graph whose nodes carry multi-view evidence bundles (geometry-aware keyframe selection, full-sequence consolidation, relation-aware refinement); generation synthesizes an occlusion-free image per instance and lifts it into a 3D asset; and GizmoAct, a VLM policy, places each asset through multi-turn GUI interaction with a 3D gizmo, emitting incremental 9-DoF edits (including 24 axis-permutation coarse rotations) and deciding itself when to stop, trained by SFT on synthetic trajectories with error injection followed by GRPO. On the authors' new R2S benchmark, scene-level detection mAP rises from 0.351 (Boxer) to 0.592, ADD-SB@0.05 improves from 79.2% to 92.0% on R2S-Object and 57.8% to 83.4% on CA-1M, and scene F-Score reaches 0.924 versus 0.794 for SAM 3D; detection gains on public CA-1M are far smaller (0.171 to 0.180). Limitations: evaluation is dominated by the self-introduced R2S benchmark with manual annotations, the pretrained VLM and image-editing model are not fully identified, no code or runtime costs are reported, and objects missed by parsing are unrecoverable.
Review complete
Assessment at a glance
5
risk flags identified
● high 0
● medium 3
● low 2
5 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- Building a scene graph whose nodes carry per-instance multi-view evidence yields the highest scene-level detection mAP in all four protocols, e.g. 0.351 to 0.592 over Boxer on R2S-Scene under the all-annotation protocol.
- GizmoAct, casting placement as multi-turn GUI interaction and trained with SFT plus GRPO, raises strict-alignment success (ADD-SB@0.05) by up to 25.6 points (57.8% to 83.4% on CA-1M) while accepting mismatched assets and coarse initial poses.
- A single general GizmoAct policy adapts to three initializers with different error profiles without retraining (Table 3).
- The composed system reaches scene F-Score 0.924 on R2S-Scene versus 0.794 for SAM 3D and 0.351 for SceneGen.
- RL over SFT mainly buys recovery from large initialization errors, cutting geodesic rotation error from 45.19 to 20.98 degrees on the 110-object hard subset.
Strengths
Placement recast as closed-loop GUI interaction is a genuine formulation contribution
Sec. 2.3 defines a gizmo-frame action space where translation/scale are expressed in units of current object size (removing metric-scale dependence), permute_axis covers all 24 axis-aligned reorientations in one action (Fig. 4), and the policy itself decides termination — unlike render-and-compare methods that need an external convergence criterion (Sec. 4.3).
Unusually thorough evaluation and ablation program
Four evaluation angles: detection on 4 protocols (Table 1), pose on 3 datasets x 3 initializers with one general policy (Tables 2-3), system-level reconstruction (Table 4), plus ablations of every parse stage (Table 5), RL pose distribution (Table 6), and RL vs two SFT variants on a 110-object hard subset (Table 7).
Honest reporting of inconvenient results
Efficiency argument for the parser is supported
RL benefit is precisely attributed
Weaknesses
Headline gains anchored to a self-introduced, manually annotated benchmark
Input-fairness gap in the system-level comparison
Core components unnamed; no code, cost, or release statement
No uncertainty quantification anywhere
Ground-truth circularity and metric caveats in pose evaluation
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
Lucida (ByteDance Seed, with PKU/ZJU) targets composable real-to-sim scene modeling: from posed RGB(-D) video of a cluttered indoor capture, produce complete, editable per-object assets placed by 9-DoF poses and organized in a scene graph. The core reframing — keep the parse-generate-place order but demand precision only at the final closed-loop step — is argued cleanly in Sec. 1. Central claims: (1) scene-graph parsing with per-instance multi-view evidence achieves the highest detection mAP in all four protocols (Table 1), e.g. 0.351 to 0.592 over Boxer on R2S-Scene (all-annotation); (2) GizmoAct, a VLM policy casting placement as multi-turn GUI interaction, lifts ADD-SB@0.05 from 57.8% to 83.4% on CA-1M and from 79.2% to 92.0% on R2S-Object (Table 2); (3) composed scenes reach F-Score 0.924 vs 0.794 for SAM 3D and 0.351 for SceneGen (Table 4). The paper also introduces R2S, its own real-world benchmark (R2S-Scene: 9 scenes; R2S-Object: 125 objects, 64 categories).
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 5 strengths & 5 weaknesses, analyzed in detail
- 5 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What is Lucida?
Lucida is a computer-vision system from ByteDance Seed that converts posed RGB-D video of a real indoor scene into a simulation-ready, editable replica: it detects object instances from multi-view evidence, generates a complete 3D asset for each (even occluded ones, via amodal image completion), and places every asset into the scene with the correct 9-DoF pose and scale.
What is GizmoAct?
GizmoAct is the vision-language-model policy inside Lucida that handles placement. It treats alignment like operating a 3D editor: it views renderings of the scene point cloud, the asset, its bounding box, and a 3D gizmo, then emits small incremental edits (rotation, translation, anisotropic scale) and decides by itself when the alignment is good enough. It is trained with supervised finetuning on synthetic trajectories plus reinforcement learning (GRPO).
What results does the Lucida paper report?
On the authors' new R2S indoor benchmark, scene-level 3D detection mAP improves from 0.351 (Boxer baseline) to 0.592; object placement raises the strict ADD-SB@0.05 success rate from 79.2% to 92.0% on R2S-Object and from 57.8% to 83.4% on CA-1M; and full scene reconstruction reaches an F-Score of 0.924 versus 0.794 for SAM 3D and 0.351 for SceneGen.
Is code or data available for Lucida?
The paper points to a project page (lucida-r2s.github.io) but contains no code release statement. The base vision-language model and the image-editing model are not named in the paper, and the R2S benchmark's release is not explicitly promised, so independent reproduction is currently difficult.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Extraction is complete through Sec. 5 and all 116 references; all tables (1-7) and equations are intact. Only the interior text of Figures 9-12 shows garbled glyphs in the PDF extraction, which does not affect any number cited in this review.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.