How Children Design and Reason about Trustworthy AI Chatbots
Deniz Ozturk, Jiayu Li, Daksh Pratap Singh, Yasitha Rajapaksha, Fasika Melese, Bahare Riahi, Shiyan Jiang, Qiao Jin, et al.
cs.HC · cs.AI
Quick verdict
Valuable mixed-methods study of children as chatbot designers; 7 trust dimensions credible, but age/domain effects are confounded and exploratory.
Summary
The paper reports a mixed-methods study in which 115 learners aged 8-18 (76 elementary, 24 middle school, 15 high school) built 119 chatbots in LUMI, a constructionist builder exposing trust-relevant controls (confidence, transparency, formality, assertiveness sliders, rules, persona, knowledge base) plus testing tools and a hidden controlled-hallucination pipeline. Combining pre/post surveys, interviews, logs, and an LLM-judged design-fidelity rubric, the authors derive seven dimensions (18 codes) of children's trust conceptions. Younger students (10-13) set higher confidence; older students (14-18) set higher transparency; academic chatbots were calibrated more transparent and formal than hobby ones; and 10 of 34 middle-school hobby bots were deliberately wrong yet still called trustworthy, revealing a purpose-fulfillment view of trust. Mean design fidelity was 2.84/4, with purpose alignment weakest (2.47). Main limitations: the three cohorts were confounded with setting, interface, and instruction, so age effects cannot be isolated; elementary data came only via teachers; multiple chatbots per student violate independence in group comparisons; the fidelity judge was validated on a prior dataset; and no code or data are released.
Review complete
Assessment at a glance
6
risk flags identified
● high 0
● medium 4
● low 2
5 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- Children designing their own chatbots express trustworthiness through interacting configuration choices, captured by seven design dimensions and 18 codes (Table 3).
- Younger students (10-13) set significantly higher Confidence while older students (14-18) set higher Transparency, and both calibrate academic chatbots to be more transparent and formal than hobby chatbots.
- Some younger students equate trustworthiness with purpose-fulfillment: chatbots deliberately built to give wrong answers were still judged trustworthy because they reliably did what they were built to do.
- Student-created chatbots showed moderate design fidelity (M=2.84/4 over 61 bots, 1,568 turns), with Purpose Alignment the least faithfully realized dimension (M=2.47).
- AI-literacy tools should help children distinguish linguistic confidence from reliability, design fidelity from factual accuracy, and social credibility from evidence (§6.3).
Strengths
Genuinely novel framing: children as designers, not users
§1-2 motivate the gap against user-centric prior work (Van Brummelen et al. 2023; Oh et al. 2025); having children translate trust beliefs into concrete slider/rule choices makes mental models observable, a real methodological advance for child-AI trust research.
Multi-source triangulation with documented rigor
Pre/post surveys adapted from a validated instrument (α .747-.795, Appendix A), thematic codebook with κ=1.00 on the 50% double-coded subsample (§4.4), interaction logs, worksheets, teacher reflections, and a fidelity rubric with reported human-AI agreement of 82.8% and quadratic-weighted κ=0.88.
The deliberately-wrong-bot finding is analytically consequential
Concrete, checkable quantitative patterns
Unusually candid limitations section
Weaknesses
Age is hopelessly confounded with context
Non-independence of chatbots in group comparisons
Conclusion asserts an analysis that is never reported
Statistical reporting errors and unexplained scales
LLM-judged fidelity validated off-dataset; no accuracy ground truth
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
The paper positions children as designers rather than users of AI. Across three settings, 115 learners (76 elementary, grades 2-3; 24 middle school; 15 high school) built 119 chatbots in LUMI, a builder exposing four trust-relevant trait sliders (Confidence, Transparency, Formality, Assertiveness, each 1-3), rules, persona, and a knowledge base. Three contributions are claimed: (1) a seven-dimension, 18-code account of what children believe makes a chatbot trustworthy (Table 3); (2) exploratory evidence that age group and chatbot domain shape trust design across configuration logs, stated beliefs, and design fidelity; (3) design implications for AI-literacy platforms. The headline empirical finding is that 10 of 34 (29.4%) middle-school hobby chatbots were deliberately configured to give wrong or nonsense answers, yet their creators judged them trustworthy because they fulfilled their intended purpose (§5.2.1, MS-P23).
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 5 strengths & 5 weaknesses, analyzed in detail
- 6 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What did this study find about children and trustworthy AI?
115 children aged 8-18 built their own chatbots in a tool called LUMI. Younger children (10-13) configured their bots to sound more confident and tied trust to friendliness and fulfilling the bot's purpose; teens (14-18) emphasized transparency and bots admitting their limits. Academic chatbots were deliberately made more formal and transparent than hobby chatbots, and some children built bots designed to give wrong answers yet still called them trustworthy because the bots did what they were built to do.
What is the LUMI chatbot-building environment?
LUMI is a constructionist tool where children configure a chatbot using sliders for confidence, transparency, formality, and assertiveness (each low/medium/high), plus rules, persona, and uploaded knowledge sources. They test it against multiple LLMs, inspect reasoning and a confidence bar, use fact-check and model-comparison tools, and share bots for peer testing. A research-only hidden mechanism could rewrite replies into plausible but incorrect answers so children would encounter hallucinations. An elementary variant, ToyTalk, adds voice interaction.
Why did some children trust a chatbot designed to give wrong answers?
Because it reliably fulfilled its intended purpose. Ten of 34 middle-school hobby chatbots were intentionally configured to give incorrect or nonsense answers, and their creators still rated them trustworthy, arguing that 'they do what they're meant to do.' The authors interpret this as younger children conflating behavioral fidelity (doing as designed) with epistemic trustworthiness (being factually reliable), which motivates tools that show these as separate indicators.
How reliable are the age comparisons in the paper?
They are explicitly exploratory. Age was confounded with setting, schedule, interface version, instruction, and data collection; elementary children were never interviewed or surveyed directly about trust; and some statistical tests treated chatbots rather than students as independent observations. The authors themselves state these patterns should be read as hypotheses for future controlled research, not established developmental effects.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Figures 1-6 (LUMI/ToyTalk interfaces, hallucination pipeline, trait distributions, survey plot) could not be analyzed from the extracted text.
- Appendix labeling is inconsistent in the extraction (survey labeled both A and B; 'Appendix 5' referenced for the rubric), and minor template artifacts (e.g., journal dates under reference [27]) suggest extraction noise.
- Review is based solely on the embedded text; supplementary instruments, the actual codebook file, and judge prompts were not available.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.