IRWOZ 2.0: A Large Language Model-driven Dialogue Dataset for Industrial Robot Conversations
Chen Li, Dimitrios Chrysostomou
cs.AI
Quick verdict
Claims LLM-built IRWOZ 2.0 (390 dialogues) fixes IRWOZ noise; BLEU gains are real but likely reflect LLM-text homogeneity, not better state tracking.
Summary
The authors present IRWOZ 2.0, an expanded industrial human-robot interaction dialogue dataset built by correcting the noisy original IRWOZ and generating new dialogues with Mistral and Claude-3.5. Corrections target three documented error types (Table 1: 4.3% typos, 2.2% incomplete markups, 3.2% mis-annotations) via manual slot/response verification and prompt-based typo removal; generation uses structured prompts encoding dataset structure, database tables, and behavioral constraints. The result is 390 dialogues across four domains (Assembly, Delivery, Position, Relocation), about 55% single-turn, publicly released on IEEE DataPort. Fine-tuning GPT-2, GPT-2-medium, and GPT-2-large on both datasets, the authors report large BLEU gains (base GPT-2 BLEU-4 0.1651 to 0.5604) but mixed state-tracking results: slot accuracy and joint goal accuracy drop for the medium (JGA 0.925 to 0.855) and large (0.938 to 0.871) models. A 23-rater human evaluation scores generated dialogues roughly 7.4-8.7 on accuracy, domain knowledge, engagement, and human-likeness. Main limitations: benchmarks run on LLM-generated text (Sec. 5.1), so BLEU gains plausibly reflect machine-text homogeneity rather than annotation quality; the abstract's state-tracking claim rests on BLEU, a generation metric; no statistical tests, no inter-rater agreement, unclear rater-to-dialogue assignment; training protocol (splits, hyperparameters) unreported; and the '3.1x faster' claim lacks any measurement.
Review complete
Assessment at a glance
6
risk flags identified
● high 2
● medium 2
● low 2
5 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- IRWOZ 2.0 expands the industrial HRI dialogue dataset to 390 dialogues across four domains via Mistral/Claude-3.5 generation plus manual and automated correction.
- Hybrid quality control reduces the original dataset's three error types, measured at 4.3% typos, 2.2% incomplete markups, and 3.2% mis-annotations (Table 1).
- Fine-tuned GPT-2 models achieve far higher BLEU on IRWOZ 2.0 (base model BLEU-4 0.1651 to 0.5604), which the authors attribute to improved dataset quality.
- Dialogue state tracking 'significantly improves', though Table 3 shows slot accuracy and joint goal accuracy regress for gpt2-medium and gpt2-large.
- A 23-rater human evaluation rates the LLM-generated dialogues approximately 7.4-8.7 across accuracy, knowledge, engagement, and human-likeness.
Strengths
Real gap addressed with a released artifact
Industrial-domain task-oriented dialogue data is genuinely scarce (Sec. 2.1 notes MultiWOZ/SGD cover travel/hospitality); IRWOZ 2.0 provides 390 annotated dialogues with belief states, system actions and small-talk labels across four concrete domains, released on IEEE DataPort with the original on GitHub (Data availability).
Quantified, transparent error taxonomy of the predecessor
Sec. 3.1/3.2 and Table 1 give per-type error rates on the original IRWOZ (typos 4.3%, incomplete markups 2.2%, mis-annotations 3.2%) instead of vague 'noise' claims, with worked examples in Fig. 2.
Published prompts for both correction and generation
Benchmark table reports unfavorable metrics too
Human evaluation with domain-relevant raters
Weaknesses
Headline claim conflates generation quality with state tracking
Evaluation-on-own-generation confound
No statistical reporting anywhere
Methodological imprecision and missing protocol
Small, low-complexity corpus limits the conclusions
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
IRWOZ 2.0 - Structured Preliminary Review
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 5 strengths & 5 weaknesses, analyzed in detail
- 6 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What is IRWOZ 2.0?
IRWOZ 2.0 is an improved dialogue dataset for industrial human-robot interaction: 390 Wizard-of-Oz-style dialogues across four industrial domains (Assembly, Delivery, Position, Relocation), created by correcting errors in the original IRWOZ dataset and generating new dialogues with the Mistral and Claude-3.5 language models. It is publicly released via IEEE DataPort for non-commercial use.
How much better is IRWOZ 2.0 than the original IRWOZ?
On language-generation metrics the gains are large: fine-tuned base GPT-2's BLEU-4 rises from 0.1651 to 0.5604. However, on the actual state-tracking metrics, slot accuracy and joint goal accuracy slightly decrease for GPT-2-medium and GPT-2-large (e.g., JGA 0.938 to 0.871), so the improvement is mainly in language quality, not state tracking. Note the benchmark evaluates on LLM-generated dialogues, which itself favors BLEU.
Where can I download the IRWOZ 2.0 dataset?
From IEEE DataPort (linked in the paper) under a non-commercial license; the original IRWOZ dataset is available on GitHub at lcroy/ToD4IR. The paper also publishes the prompts used for typo correction (Appendix A) and dialogue generation (Figure 4).
How was IRWOZ 2.0 created and validated?
In three steps: (1) manual verification of slot values and responses in the original data, fixing mis-annotations (3.2% of turns) and incomplete markups (2.2%); (2) Claude-3.5 prompt-based removal of typos (4.3% of turns); (3) generation of new dialogues from structured prompts specifying dataset structure, database content and behavioral rules. Quality was then checked by fine-tuning GPT-2 models and by 23 human raters scoring sampled dialogues on accuracy, domain knowledge, engagement, and human-likeness (scores about 7.4-8.7).
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- All figures (Fig. 1-6, B2, B3, D4, D5) are images in the PDF extraction; their contents (e.g., the full prompt in Fig. 4, score distributions in Fig. 6, generated dialogue samples in Appendices C-D) could only be assessed via captions and numbers quoted in the body text.
- The extraction's final line ends with a garbled page artifact ('...domainsains 21') at the end of Appendix D; all sections, Tables 1-3, Table B1 and the reference list up to that point appear complete, so no substantive truncation is assumed.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.