Beyond AI Literacy: A Structured Review and Exploratory Meta-Analysis of Measures for Competent Generative-AI Use
cs.HC · cs.AI
Quick verdict
Reviews 24 studies of GenAI-literacy measures, pooling 3 same-program correlations (r=.055, CI incl. 0); candid but thin evidence, battery unvalidated.
Summary
This structured review extends the 2024 COSMIN-based inventory of AI-literacy scales with a targeted search through 17 August 2026, coding 24 focal empirical publications into four measurement domains: knowledge and use, epistemic oversight, reliance calibration, and operational control of tool-using agents. An exploratory random-effects meta-analysis pools the only three directly reported same-sample subjective-objective correlations, all from one research program (Koch 2024 r=.21, N=120; AICOS 2025 r=.04, N=514; AICOS-S 2026 r=.05, N=2,131), yielding REML r=.055 with a Hartung-Knapp 95% CI of [-.047, .156] on combined reported N=2,765. The largest study has an unresolved discrepancy between its reported r and p-value and carries 77.2% of the weight; adding a synthetic composite of 12 cross-factor correlations from a fourth study gives r=.079, CI [-.025, .181]. The author concludes this evidence cannot establish a population correlation, validate workplace cutoffs, or support substituting self-ratings for performance scores. The descriptive synthesis catalogues objective tests (AICOS-S, GLAT), oversight measures (Critical Thinking in AI Use Scale, CAILS, AILIS), reliance, trust, and dependency scales, and finds no validated instrument covering the full set of agent-control behaviors (scope, permissions, recovery, state isolation, independent review, evidence-based closure). A four-layer workplace battery with non-compensatory rules is proposed strictly as a design for validation. Key limitations: a non-reproducible seeded search, single-coder coverage ratings, a one-program primary pool, and no accompanying data or analysis scripts.
Review complete
Assessment at a glance
5
risk flags identified
● high 0
● medium 3
● low 2
5 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- Post-2024 work has substantially expanded objective AI-literacy measurement (GLAT, AICOS/AICOS-S, Klein-Avraham et al.), reversing the self-report dominance recorded in the 2024 COSMIN baseline.
- In the only three same-sample comparisons available, all from one research program, perceived and demonstrated AI literacy correlate near zero (pooled REML r=.055, Hartung-Knapp 95% CI [-.047, .156], N=2,765).
- No validated individual-level instrument in the focal corpus tests the full combination of agent scope, permissions, recovery, state isolation, independent review, and evidence-based closure.
- A four-layer workplace battery (objective foundation, epistemic oversight, reliance, situated operational challenge) with non-compensatory rules is proposed strictly as a design awaiting validation.
- Trust, dependency, and reliance are adjacent constructs that must be kept separate from competence measurement.
Strengths
Exceptionally candid uncertainty reporting
The paper foregrounds its own weaknesses: the unresolved AICOS-S r/p discrepancy (r=.05, p=.11 vs computed p≈.021), the 77.2% weight concentration, REML τ² at the zero boundary, and an explicit refusal to impute replacement values (Sections 3.3, 4.3, Table 2 note).
Small-k meta-analytic best practice
Fisher-z REML with Paule-Mandel refit, Hartung-Knapp primary intervals with normal-theory comparison, prediction intervals flagged as unreliable at τ²=0, plus leave-one-out and program-aggregation sensitivity analyses all tabulated (Table 4).
Timely, explicitly bounded coverage-gap finding
Useful construct separation
Concrete validation roadmap
Weaknesses
The meta-analysis adds little beyond its constituent studies
The search cannot be reproduced or audited
Single-coder interpretive coverage map
The proposed battery is entirely unvalidated
No data or code artifacts
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Beyond AI Literacy — Preliminary Review
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 5 strengths & 5 weaknesses, analyzed in detail
- 5 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
How well do self-reported AI literacy scores align with objective AI literacy test performance?
According to this paper's exploratory meta-analysis, barely at all. The only three published same-sample correlations — all from one research program using the MAILS self-report family — pool to r = .055 (Hartung-Knapp 95% CI [-.047, .156], combined N = 2,765), statistically indistinguishable from zero. Adding a fourth study's composite of cross-factor correlations raises the estimate to r = .079 (CI [-.025, .181]), still including zero. The evidence base is too small to establish a population correlation, but it provides no support for treating self-ratings as interchangeable with performance scores.
What are the four domains of competent generative-AI use proposed in this paper?
The paper organizes measurement into four domains: (1) knowledge and use — perceived-capability scales (AILS, SNAIL, MAILS) and objective tests (AILIT, GLAT, AICOS, AICOS-S); (2) epistemic oversight — verification of AI-generated claims (Critical Thinking in AI Use Scale, AILIS, CAILS); (3) reliance calibration — reflective vs. thoughtless reliance behaviors, with trust and dependency treated as separate adjacent constructs; and (4) operational control of tool-using agents — permissions, recovery, state isolation, independent review, and evidence-based closure, a domain for which no validated individual-level instrument exists yet.
Is there a validated test that measures how well a person supervises AI agents?
No. The review finds no validated individual-level instrument in its corpus that tests the full combination of agent scope and permissions, plan invalidation and recovery, state isolation, separation of maker and reviewer roles, and evidence-based closure. Related work exists — Dorneich et al.'s team-level behavior metrics and Bogart et al.'s context-specific scenario assessment — but the paper proposes 'agentic operational competence' only as a hypothesis for future instrument development, not an established construct.
Which AI-literacy tests does the paper recommend for workplace assessment?
It proposes (but does not validate) a four-layer battery: AICOS-S (12 objective items with German adult norms) or GLAT (20 items, higher-education samples, with criterion evidence) for foundation knowledge; the Critical Thinking in AI Use Scale plus a short verification task for oversight; the GenAI Reliance Behaviors Scale for reliance patterns; and a role-specific situated task for operational control — assessed with non-compensatory rules, each threshold requiring local validation before placement use.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Figures 1-4 are described only via captions and reported numbers; the forest plot's visual interval display could not be independently inspected.
- This review inherits the paper's own admission that the seeded search is not reproducible, so corpus completeness beyond stated paths cannot be assessed from the text.
- The paper is an arXiv preprint citing several 2026 instruments; cited psychometric details (e.g., AICOS-S CFI=.995) were taken as reported and not verified against sources.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.