Generative AI and Extended Reality in Collaborative Architectural Design Education: An Exploratory Studio Study
Yao Xiao, Max Chen, Yichen Li, Nathaniel Powers, Maxwell Wiesenfeld, Gillian Smith, Soroush Farzin, Shichao Liu
cs.HC · cs.AI
Quick verdict
Classroom GenAI+XR study reports honest mixed effects, but 7-team sample, participant-level clustering, and an impossible n=4 p-value weaken causal inferences.
Summary
An exploratory mixed-methods classroom study at Worcester Polytechnic Institute embedded GenARch — a pipeline chaining DALL·E 3 image generation, Shap-E/TripoSR/Tripo3D 2D-to-3D conversion, and multi-user Meta Quest 3/Arkio review — into one undergraduate architectural design studio. Twenty-seven students in seven self-selected teams were assigned at team level: four teams (N=16) used GenARch during a one-week conceptual-design phase while three teams (N=11) continued the usual workflow. Pre/post surveys (design self-efficacy, collaborative-learning attitudes, Team-Q), a blinded seven-rater panel evaluation of presentations, and group interviews supplied the data. Difference-in-differences estimates showed larger relative declines in self-efficacy confidence (β=−1.675, p=0.021) and outcome expectancy (β=−2.088, p=0.002) for GenARch teams, a positive estimate for Team-Q conflict management (β=0.479, p=0.014), and no significant panel-score differences (GenARch median 3.45 vs Control 3.87), with GenARch lower on all five criteria. Interviews portrayed complementary roles — GenAI for rapidly externalizing tentative ideas, XR for scale and site evaluation — plus frictions around control, dimensional fidelity, shared attention, and motion sickness. Limitations include only seven teams with participant-level clustered standard errors despite team-level assignment, ten uncorrected DiD tests, a leave-one-out rater adjustment that maximizes agreement, a Spearman r=1.00 with an impossible p<0.001 on four teams, and low original inter-rater agreement (Kendall's W 0.234–0.466).
Review complete
Assessment at a glance
6
risk flags identified
● high 2
● medium 2
● low 2
5 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- Embedding GenARch in the studio workflow was associated with larger relative pre–post declines in design self-efficacy confidence and outcome expectancy than control (DiD β=−1.675, p=0.021; β=−2.088, p=0.002).
- The GenARch condition showed a positive DiD estimate for Team-Q perceived conflict management (β=0.479, p=0.014) with no other significant teamwork or attitude changes.
- Blinded seven-rater panel scores of final presentations did not differ significantly between conditions (overall median 3.45 vs 3.87), with GenARch teams lower on all five criteria.
- Students used GenAI and XR complementarily across design stages — GenAI for externalizing tentative ideas, XR for scale/site/spatial evaluation — while facing control, fidelity, shared-attention, and comfort challenges.
Strengths
Authentic course integration
Deployed inside a live sophomore studio as a technology probe (§3.1–3.2) with IRB approval, team-level randomization of intact teams, and baseline comparability checks using exact/non-parametric tests (§3.4) — rare ecological validity for GenAI/XR education studies.
Candid mixed-results reporting
The paper foregrounds negative findings: relative declines in confidence and outcome expectancy (Table 2), lower GenARch medians on all five panel criteria (Fig. 4a: 3.45 vs 3.87 overall), and consistently hedged inferential language (§4.2, §6).
Transparent statistical diagnostics
Actionable qualitative strand
Concrete design implications
Weaknesses
Team-level treatment, participant-level inference
Impossible p-value
Data-dependent rater adjustment
No multiplicity control and weak scales
Underpowered null on outcomes
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Preliminary Review — GenARch: GenAI + XR in Collaborative Architectural Design Education
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 5 strengths & 5 weaknesses, analyzed in detail
- 6 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
Did GenAI and XR tools improve architecture students' design outcomes in this study?
No. A blinded seven-member panel rated the GenARch teams' final presentations lower than control teams' on all five criteria (overall median 3.45 vs 3.87 out of 5), but the difference was not statistically significant with only seven teams.
What is GenARch?
GenARch is a pipeline the authors built that connects DALL·E 3 text-to-image generation, 2D-to-3D conversion via Shap-E, TripoSR, or Tripo3D, and shared review of the resulting models in the multi-user VR app Arkio on Meta Quest 3 headsets, which architecture students used for conceptual ideation.
Did using GenAI and VR make students less confident?
The GenAI+XR group showed larger relative pre-to-post declines in design self-efficacy confidence and outcome expectancy than controls (difference-in-differences p=0.021 and p=0.002). The authors suggest hands-on exposure made the tools' limitations visible, but the study cannot confirm why.
Is the study data available?
Anonymized qualitative coding materials, the evaluation rubric, and the interview protocol are on OSF; participant-level survey data are restricted to protect confidentiality, with access possible by contacting the authors.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Supplementary Appendices A–C (rubric, interview protocol, experimental details) are referenced but their contents are not part of the provided text; rubric-level judgments rely on the authors' in-text descriptions.
- Figures 1–4 are images not recoverable from extraction, and Fig. 3's caption region contains garbled placeholder characters (xxxxxx), so generated-output quality could not be visually assessed.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.