The Vibe Shift in Software Engineering: Evaluating AI-Led Conversational Programming for Performance, Cognition, and Responsible Adoption
Sales G. Aribe, Louie Jay S. Labastida
cs.SE · cs.AI · cs.ET
Quick verdict
Claims 30-participant study finds vibe coding 27% faster but with lower maintainability and security; direction plausible, statistical reporting too thin.
Summary
The paper reports a mixed-methods, quasi-experimental comparison of three programming modalities — traditional coding, AI-assisted coding with GitHub Copilot, and a 'vibe coding' condition using ChatGPT-style conversational LLM interaction — with 30 Philippine participants (15 professional developers, 15 advanced IT students) completing three standardized Python/JavaScript tasks under all conditions. Quantitatively, vibe coding cut task completion time by 27% versus traditional coding and 12% versus Copilot-assisted work, with significant ANOVA results (p < .05) reported for time, debugging iterations, maintainability index (≈60 vs ≈75/70), cyclomatic complexity, and security vulnerabilities (+22% versus manual code). SUS scored 71.4 for vibe coding (vs 74.6 AI-assisted, 65.8 traditional); NASA-TLX workload was 55.5 (between AI-assisted 53.8 and traditional 64.0). Thematic analysis of interviews yielded four themes — trust calibration, loss of control, cognitive adaptation, and prompt-engineering strategy — and the authors propose a three-pillar responsible-adoption framework. Major limitations: statistical reporting omits F statistics, effect sizes, and dispersion despite promising η²; no counterbalancing against learning/carryover effects across repeated tasks; confound between interaction paradigm and tool; relative percentages lack absolute baselines; and qualitative coding lacks inter-rater reliability. No code or data availability is stated.
Review complete
Assessment at a glance
6
risk flags identified
● high 2
● medium 3
● low 1
5 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- Vibe coding cut task completion time by 27% versus traditional coding and 12% versus Copilot-assisted coding (repeated-measures ANOVA, p < .05).
- The speed gain trades off against quality: maintainability index fell to ≈60 (vs ≈75 traditional, ≈70 AI-assisted) and security vulnerabilities rose ≈22% over manual code.
- Cognitive workload is redistributed rather than removed (NASA-TLX 55.5 vs 53.8 AI-assisted and 64.0 traditional): less syntax recall, more prompt formulation and linguistic reasoning.
- Developer experience coheres around four constructs — trust calibration, loss of control, cognitive adaptation, and prompt-engineering strategy — with perceived loss of control framed as a security-risk amplifier.
- Responsible adoption requires a three-pillar framework: hybrid integration, human oversight with transparent accountability, and context-aware (risk-tiered) deployment.
Strengths
Genuine construct differentiation with a clear gap statement
Sec. I and I-F distinguish token-level assistance (Copilot) from intent-driven conversational generation and motivate vibe coding as a distinct, under-examined paradigm rather than rehashing Copilot usability studies.
Triangulated mixed-methods architecture
Convergent parallel design (Sec. II-A, II-G) merges ANOVA performance metrics, SUS/NASA-TLX instruments, and Braun & Clarke thematic analysis; themes (Table 5) are explicitly checked against quantitative results, e.g. the 67% 'only as a starting point' figure against trust calibration.
Standardized tasks and prompt scaffold
Balanced reporting of trade-offs
Validated instruments and ethics documentation
Weaknesses
Inferential statistics essentially unreported
Order effects unaddressed
Paradigm–tool–model confound
Relative percentages without absolute anchors
Qualitative rigor gaps and editorial defects
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
The paper (IJASEIT Vol. 16 No. 4, 2026; arXiv 2609.09560) evaluates 'vibe coding' — fully conversational LLM-driven programming — against traditional coding and Copilot-style AI-assisted coding on three axes: performance/quality, usability/cognition, and responsible adoption. Central claims: (1) 27% faster than traditional and 12% faster than AI-assisted coding (p < .05); (2) quality trade-offs — maintainability index ≈60 vs ≈75 (traditional) and ≈70 (AI-assisted), +22% security vulnerabilities vs manual code; (3) workload shifts from syntax to prompt formulation (NASA-TLX 55.5, between 53.8 and 64.0); (4) four qualitative constructs (trust calibration, loss of control, cognitive adaptation, prompt-engineering strategy); (5) a three-pillar responsible-adoption framework. Claims 1–3 are empirically interesting; claims 4–5 are interpretive.
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 5 strengths & 5 weaknesses, analyzed in detail
- 6 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What is vibe coding according to this study?
The study defines vibe coding as an AI-led conversational programming paradigm where developers express high-level intent in natural-language prompts and an LLM generates complete functional code — unlike inline assistants such as GitHub Copilot, where developers retain token-level control. The term was coined by Karpathy in 2025.
How much faster is vibe coding than normal coding in this experiment?
In this 30-participant study, participants completed standardized Python/JavaScript tasks 27% faster under the vibe-coding condition than with traditional manual coding, and 12% faster than when using GitHub Copilot's inline suggestions (repeated-measures ANOVA, p < .05 reported).
What are the downsides of vibe coding found in the paper?
The paper reports lower code maintainability (maintainability index ≈60 vs ≈75 for traditional coding), higher cyclomatic complexity, roughly 22% more security vulnerabilities than manually written code, and moderate cognitive workload (NASA-TLX 55.5) driven by prompt engineering and verification of opaque AI output.
What framework does the paper recommend for adopting vibe coding?
A three-pillar responsible-adoption framework: (1) hybrid integration of human and AI capabilities (AI for prototyping/scaffolding, humans for review and optimization), (2) human oversight with transparent accountability (peer review, static analysis, audit logs), and (3) context-aware deployment with graduated risk tiers (exploratory use, assisted production, restricted domains).
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Figures 1–4 are raster images; only their captions and adjacent narrative were extractable (some Fig. 2 axis numbers leaked into text), so diagram-content judgments rely on in-text descriptions.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.