DataCanvas-EDU: An Agentic Framework for Instructor-Guided Synthetic Data Generation in Business Analytics Education
Bang An, Maria Hamdani, Joseph Fox
cs.HC · cs.AI
Quick verdict
Agentic framework for instructor-guided synthetic data in analytics teaching; honest and reusable, but unevaluated — no user study, and demo case has flaws.
Summary
The paper presents DataCanvas-EDU, an agentic framework in which an instructor specifies a business scenario, learner profile, and intended statistical patterns through conversation with an LLM agent; the agent then generates a synthetic dataset from scratch, numerically verifies that each designed pattern is present in the exported records, and produces assignments, reference solutions, and rubrics. The workflow is organized into four phases (Plan, Create, Verify/Test Analysis, Evaluate) and implemented as a versioned AI Agent Skill (v0.1.3) installable in Claude, Claude Code, and compatible Codex/ChatGPT environments, with code on GitHub. The motivation combines the inflexibility of real data with training-data contamination of standard teaching datasets, citing memorization of Titanic and Iris. The demonstration is WindowDash, a fictional food-delivery case with 15,000 orders and nine designed patterns, each tied to generation rules, share-based comparisons with minimum support, and rubric criteria. Notably, the authors candidly report weaknesses in their own artifact: the Price-Rating pattern rests on three orders (mean rating 2.0), and the verification program checks the office-lunch pattern only on Mondays/Tuesdays while the generator uses all weekdays, omitting the age pattern from printed checks. The main limitations are the complete absence of empirical evaluation — no instructor or student use, no preparation-time measurements — a demonstration restricted to one case and one CSV table, and an untested contamination rationale. The paper is best read as a design contribution with a released tool, not a validated educational intervention.
Review complete
Assessment at a glance
5
risk flags identified
● high 0
● medium 3
● low 2
5 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- An instructor can, through conversation, specify teaching goals and intended patterns that an LLM agent turns into a verified synthetic dataset plus assignment, reference solution, and rubric.
- Carrying each pattern's definition through generation, verification, reference analysis, and assessment reduces the repeated work of preparing and revising teaching cases.
- Freshly generated cases with checked reference answers sidestep the training-data contamination that afflicts widely used public teaching datasets.
- The framework is practically usable, demonstrated by WindowDash (15,000 orders, nine designed patterns) and released as installable Agent Skill v0.1.3.
Strengths
Transparent self-critique
Sections 6.3-6.4 disclose the artifact's own defects — an n=3 Price-Rating pattern (mean rating 2.0) and verifier/generator mismatches (office-lunch checked Mon/Tue only; age check omitted) — a candor rare in tool papers.
End-to-end traceability per pattern
The case specification (Table 1) binds each pattern to its generation rule, measurement (Eq. 1 difference-in-proportions with acceptance range and minimum support), reference evidence, and rubric line; Section 4.3 insists generator probabilities are not evidence until measured on exported records.
Released, versioned, reproducible artifact
Pedagogical grounding
Contamination-relevant task design
Weaknesses
No empirical evaluation
Flagship demo ships unreconciled
Inadequately supported pattern accepted
Contamination rationale untested in situ
Narrow generality
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
DataCanvas-EDU is a framework paper with a released tool: an AI Agent Skill (v0.1.3, GitHub) that turns a conversational teaching brief into a synthetic dataset, reference analyses, assignments, and rubrics. The workflow has four phases — Plan, Create, Verify/Test Analysis, Evaluate — anchored by an executable case specification (`specification.json`) recording objectives, schema, patterns, checks, and review decisions (Section 4.2). Three contributions are claimed: instructor-guided dataset design, a coupled data-and-materials workflow, and a reusable implementation demonstrated on WindowDash (15,000 orders, nine patterns). The motivating problems are the inflexibility of real data and training-data contamination of public teaching datasets (Bordt et al. on memorization of Titanic/Iris rows; Sections 1 and 2.3). Claims are hedged in the abstract (the framework is *intended to* simplify case preparation), but the paper measures none of its benefits.
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 5 strengths & 5 weaknesses, analyzed in detail
- 5 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What is DataCanvas-EDU?
A framework, released as an installable AI Agent Skill (version 0.1.3), in which an instructor converses with an LLM agent to design a synthetic teaching dataset: the instructor specifies the business scenario and the statistical patterns students should discover, the agent generates the data with a recorded random seed, verifies each pattern numerically on the exported records, and produces assignments, reference solutions, and grading rubrics. Code and installation instructions are on GitHub.
Does DataCanvas-EDU solve LLM training-data contamination in analytics courses?
It addresses the concern by generating new datasets from scratch, so public worked analyses of standard datasets like Titanic or Iris do not match the assigned case. However, the paper presents no experiment showing this changes student or LLM behavior — the contamination rationale is argued from prior literature (e.g., memorization of Titanic and Iris), not tested by the authors.
What is the WindowDash case?
The paper's demonstration dataset: a fictional food-delivery scenario with 15,000 orders and nine intentionally planted patterns (for example, a pickup preference in the office district at weekday lunch, and lower ratings for expensive slow deliveries), used in an undergraduate assignment where students must find and justify at least five supported findings. The authors themselves note one pattern rests on only three orders and that some verification checks were inconsistent with the generator.
Has the framework been tested with real instructors or students?
No. The paper reports no user study, no preparation-time measurements, and no learning outcomes; a healthcare example appears only as a short planning-conversation snippet. The authors state that studies of preparation time, revision effort, and material quality are future work.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Figures 1-6 are available only as extracted text and axis labels; the visual quality of the reference charts could not be assessed.
- The public repository (github.com/BANG23333/datacanvas-edu) was not inspected; statements about the Skill's implementation rest on the paper text alone.
- Minor spacing/extraction artifacts in the embedded text; section numbering and reference formatting may differ slightly from the PDF.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.