Zero-Cost Digital Automation for Scalable Assessment in a National Mechanical Ventilation Curriculum
Ahluwalia, A. K., Tran, K. N., Staitieh, B. S., Keller, M. B., Sun, J., Lee, B. W., Seam, N., For the Critical Care Education Research Consortium,
medical education
Quick verdict
Useful pre-post study: automated Google tracking and grading accompanied higher portfolio submission in a national MV program; effects remain confounded.
Summary
This preprint reports a pre-post implementation study of a zero-cost digital infrastructure supporting a national mechanical ventilation (MV) preceptorial program for pulmonary/critical care fellows and attendings. The authors built two tools on the free Google ecosystem—Google Sheets plus Apps Script for automated waveform-portfolio (WP) tracking and reminder emails to non-submitters, and Google Forms plus rule-based scripting for automated grading of the case-based waveform examination (CBWE), operationalizing an expert-defined partial-credit rubric. Comparing Fall 2024 (manual tracking; 68 learners, 19 institutions) with Fall 2025 (standardized requirements plus automation; 92 learners, 24 institutions), timely portfolio submission rose from 45.6% to 68.5% (Fisher's exact p=0.006); among submitters, fully complete portfolios rose from 29.0% to 55.6% (p=0.017) and median completion from 94.7% to 100% (Mann-Whitney p=0.003); submitted images rose 7.7-fold (194 to 1498). Manual CBWE grading averaged 5.1 minutes per exam, projecting 8.5 faculty-hours per grader per 100 exams that automation would eliminate; retrospective automated grading of past exams matched adjudicated scores, but no quantitative agreement statistics are reported. The authors candidly acknowledge that standardization and automation were introduced simultaneously, cohort composition changed, and outcomes measure submission logistics rather than knowledge or clinical competency; prospective deployment of the grader had not yet occurred.
Review complete
Assessment at a glance
5
risk flags identified
● high 0
● medium 2
● low 3
5 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- Standardized portfolio requirements plus automated tracking and targeted reminders were associated with timely fall portfolio submission rising from 45.6% (31/68) to 68.5% (63/92) and full completion among submitters nearly doubling from 29.0% to 55.6%.
- A rule-based Google Forms/Apps Script system can operationalize the expert-defined partial-credit CBWE rubric accurately, as shown by retrospective agreement with adjudicated manual scores.
- Automation eliminates a projected 8.5 faculty grading-hours per grader per 100 examinations (from a measured 5.1 minutes per exam) and enables immediate individualized feedback plus item-level answer-distribution analytics.
- A zero-cost Google-ecosystem infrastructure decouples administrative scalability from scarce expert faculty time, offering a replicable model for other competency-based training programs.
Strengths
Candid, unusually complete limitations section
The authors state outright that standardization and automation cannot be disentangled, that enrollment growth and program maturity may confound the pre-post contrast, and that the outcome is submission logistics rather than competency — matching the Data Analysis framing of between-cohort comparisons as 'descriptive associations'.
Concrete, replicable technical blueprint
Components are named and specified: Google Sheets/Apps Script tracker with weekly reminders targeted only at non-submitters and aggregate leadership digests; Google Forms rule-based CBWE grader implementing the consensus full/partial/penalty rubric; 100% cap on completion scoring; metadata stripping plus expert review and cropping of waveform images before storage or reuse.
Quantified workload baseline rather than anecdote
Triangulated formative outcomes with appropriate tests
Transparency about AI-assisted development
Weaknesses
Attribution is impossible by design
Automated-grading accuracy is asserted, not quantified
Outcome metrics changed with the intervention
Process outcomes, no learning endpoint
Selection and cohort drift between years
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
This medRxiv preprint (posted September 17, 2026; not peer reviewed) reports a pre-post implementation study in medical education. The authors — a PCCM/CCM educator consortium — describe two digital tools built entirely on free Google infrastructure to support a national mechanical ventilation (MV) preceptorial program: (1) a waveform portfolio (WP) tracking and reminder system (Google Sheets + Apps Script) and (2) an automated, rule-based grader for the case-based waveform examination (CBWE) (Google Forms + Apps Script), operationalizing an expert consensus partial-credit rubric that includes penalties for potentially harmful management choices.
Four central claims: (i) timely fall WP submission rose from 45.6% (31/68) in Fall 2024 to 68.5% (63/92) in Fall 2025; (ii) among submitters, fully complete portfolios rose from 29.0% (9/31) to 55.6% (35/63), with median completion shifting from 94.7% (IQR 57.9–100) to 100% (IQR 91.7–100); (iii) automated grading reproduces adjudicated manual scores while eliminating a projected 8.5 faculty-hours per grader per 100 exams (from a measured 5.1 ± 0.59 minutes per exam) and adding item-level analytics plus immediate learner feedback; (iv) the model is replicable in other resource-constrained competency-based programs. The Data Analysis section frames between-cohort contrasts as 'descriptive associations' — an appropriate hedge that the title nonetheless stretches.
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 5 strengths & 5 weaknesses, analyzed in detail
- 5 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What did this study find about automated tracking in medical education?
In a national mechanical ventilation preceptorial program, switching from manual tracking to automated Google Sheets-based portfolio tracking with standardized requirements and targeted reminders was associated with timely fall submission rising from 45.6% (31/68 learners) in 2024 to 68.5% (63/92) in 2025, and full portfolio completion among submitters nearly doubling from 29.0% to 55.6%. Because standardization and automation launched together, the gains cannot be attributed to automation alone.
Is the software open source — where is the code?
The paper describes a tracker built on Google Sheets plus Apps Script and an exam grader built on Google Forms plus rule-based scripting, drafted with help from ChatGPT 4o/5 and Gemini 2.5/3 and documented in supplemental material. No public code repository or data availability statement appears in the preprint text.
Did automation improve learning outcomes or competency?
No. All outcomes measure assessment logistics — submission rates, completeness, grading time — not knowledge retention, waveform interpretation accuracy, or clinical competency. The authors state that future work should test whether improved completion translates into measurable ventilator-management competence.
How much faculty grading time does automated CBWE grading save?
Manual grading averaged 5.1 minutes per examination (SD 0.59), based on two expert graders timing six stratified past exams. The authors project about 8.5 hours per grader per 100 exams, which rule-based automated grading would eliminate; prospective deployment was planned for Summer 2026.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Figures 1 and Supplemental Figures 1-3 are caption-only in the extracted text; the rubric-to-rules conversion example and score report layouts could not be inspected.
- The abstract in the metadata is truncated; this review is based on the embedded full text, which appears substantially complete through the references and Table 1.
- No data or code availability statement is visible in the extracted text; its absence may be an extraction artifact rather than an omission.
- This is a medRxiv preprint posted September 17, 2026 and not certified by peer review.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.