KnowDemo: Knowledge-Guided Robot Demonstration Generation from Human Videos
Zhiyuan Gao, Yanxiang Zhan, Mohammad Khoshnazar, Jeroen Schäfer, Michael Beetz
cs.RO · cs.AI
Quick verdict
KnowDemo uses VLM-extracted task knowledge to generate diverse robot demos with sim-to-real π0.5 transfer; promising, but small-scale with limited baselines.
Summary
KnowDemo is a framework for generating robot manipulation demonstrations from a single human video, aimed at a specific target workspace. A digital twin of the workspace is built in Isaac Sim from segmented and reconstructed assets (SAM3, SAM3D Objects, URDF-Anything+). A frozen VLM (Qwen3.8-27B) parses the video into subtasks and produces structured manipulation knowledge — task conditions, demonstrated contact/motion references, and permissible execution variations — encoded in CRAM designators. Resolution against target-scene entities and geometry guides candidate sampling and screening: grasp strategies, approach directions (approach-cone constraints), placement sampling, and admissible subtask orders, followed by cuRobo motion planning and Isaac Sim validation. Accepted demonstrations share the inferred task instruction and carry verified execution labels. Experiments on three tasks (drawer storage, two-fruit placement, microwave closing) show more verified execution modes than a reference-only configuration (1→2 for two tasks; contact-position variation only for the third), and a shared-backend ablation shows task-guided grasp sampling raises complete-task planning success from 29/50 to 50/50 (Cup-in-Drawer) and 23/50 to 45/50 (Cup-to-Cabinet). Fine-tuning π0.5 on 100–200 generated simulation demos per task yields 50–70% success over 20 real Franka trials per task. Main limitations: narrow task set, mode-count diversity metric against a weak reference-only baseline, no head-to-head comparison with other video-to-demo systems, small evaluation samples without confidence intervals, and no stated code release.
Review complete
Assessment at a glance
5
risk flags identified
● high 0
● medium 5
● low 0
4 strengths · 0 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- Separating required task conditions from demonstration-specific choices, via VLM-extracted CRAM-structured knowledge, enables robot demonstrations with alternative contact strategies and subtask orders from one human video.
- Grounding task knowledge in target-scene entities and geometry before planning raises complete-task candidate planning success (Table II: 29/50→50/50 and 23/50→45/50).
- Generated demonstrations yield more verified execution modes than a reference-only adaptation configuration on two of three tasks (Table I), with contact-position variation only on the third.
- π0.5 fine-tuned solely on generated simulation demonstrations transfers to a real Franka at 50–70% complete-task success over 20 trials per task (Table III).
Strengths
Clean conceptual separation of task requirements from demonstration-specific choices
Sec. III-B encodes required conditions (before/during/after phases, typed predicates) separately from observed references and permissible variations inside CRAM designators, with evidence links separating video observation from commonsense deduction — a direct answer to the contact/order-locking limitation of MimicGen/DemoGen-style adaptation.
Genuinely controlled ablation of the key mechanism
Table II isolates task-guided strategy selection within one shared backend with paired initial states and an equal 50-candidate budget, evaluated on complete-task plans before execution; failures are explained mechanistically (grasps that permit pickup but block placement or withdrawal).
Honest diversity accounting
End-to-end validation including sim-to-real
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
- All 4 strengths & 0 weaknesses, analyzed in detail
- 5 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.