Talk to Me, Jarvis: An Open-Source Edge-Deployable Voice Assistant Framework for Autonomous Racecars
Daniel Henel, Frederik Werner, Alexander Langmann, Johannes Betz
cs.LG · cs.RO
Quick verdict
Claims a fine-tuned Mistral-7B hits 97.63% command accuracy at 1.39s on a laptop GPU; plausible but tested only on synthetic LLM-augmented data.
Summary
The paper presents Jarvis, an open-source, fully offline voice assistant for issuing high-level behavioral commands to autonomous racecars. The pipeline chains openWakeWord trigger detection, local Whisper base.en speech-to-text, a text-to-command classifier, and Coqui VITS text-to-speech feedback, with operator confirmation and ROS2 integration into a racing software stack. The core contribution is a domain-specific QLoRA fine-tuning of Mistral-7B (4-bit). From an expert-curated 85-command seed set, the authors built a 1,645-sample, 17-class dataset plus out-of-scope class via GPT-4-driven augmentation, split 80/20. Zero-shot benchmarking of 15 models (Table I) showed online models (o3-mini 85.88%, GPT-4o 76.47%) more accurate but slow (up to 21.08s), while local models were fast but inaccurate (Mistral-7B 27.06% at 0.64s). After two-stage fine-tuning, Mistral-7B reached 97.63% accuracy with 1.39s average inference on an RTX 3060 laptop GPU, exceeding all evaluated cloud models. Main limitations: the test set is synthetic LLM-augmented text rather than real operator speech, so field accuracy is unknown; latency excludes ASR/TTS and is measured on different GPUs across stages; results are single-run without variance or significance testing; no compact encoder baselines (e.g., BERT-style classifiers) are compared. The authors honestly note the data limitations.
Review complete
Assessment at a glance
5
risk flags identified
● high 1
● medium 3
● low 1
5 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- A QLoRA fine-tuned Mistral-7B classifier achieves 97.63% intent-recognition accuracy with 1.39s average inference on an RTX 3060 laptop GPU (Table III).
- The fine-tuned local model outperforms all evaluated online-hosted models, whose best zero-shot accuracy was 85.88% (o3-mini) at 21.08s latency (Table I).
- Locally fine-tuned open-weight models can surpass online-hosted models in both accuracy and latency for narrow, domain-specific command classification.
- The full open-source pipeline (openWakeWord + Whisper base.en + classifier + Coqui TTS + confirmation via ROS2) operates entirely offline on consumer hardware.
Strengths
Complete, genuinely open-source edge system
Integrates openWakeWord, local Whisper base.en, LLM classifier, and Coqui VITS TTS with a confirmation loop (Section II-B, Fig. 3); the final model was fine-tuned and evaluated on a consumer RTX 3060 Laptop GPU (Section III-E), substantiating the edge-deployability claim.
Transparent fine-tuning protocol
Two-stage QLoRA pipeline with a documented grid search over learning rate, epochs, gradient accumulation, and LoRA rank (Section III-C), and full final configuration in Tables II-III (lr 5e-5, 2 epochs, r=32/alpha=32, AdamW 8-bit, bnb-4bit).
Broad zero-shot comparison with explicit trade-off framing
Candid limitations discussion
Safety-aware interaction design
Weaknesses
Synthetic test set inflates the headline accuracy
No statistical rigor
Confounded latency comparisons
Missing decisive baselines; zero-shot local numbers look artifactual
Bookkeeping inconsistencies and unstated dataset release
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
This is a system/method paper (accepted to ITSC 2026) presenting Jarvis, an offline voice assistant through which a race engineer issues high-level behavioral commands (start/stop, speed targets, pit entry/exit) to an autonomous racecar. The claimed contributions (Section I-A) are: (1) a latency-versus-accuracy comparison of online-hosted and local LLMs for command classification (Table I, 15 models); (2) a fine-tuning and deployment pipeline yielding a local model that beats online alternatives on both axes; (3) an open-source architecture integrating wake-word detection, Whisper transcription, LLM classification, and TTS, demonstrated on consumer laptop hardware. The headline numbers are 97.63% intent-recognition accuracy and 1.39 s average inference (abstract; Table III).
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 5 strengths & 5 weaknesses, analyzed in detail
- 5 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What accuracy does the Jarvis voice assistant achieve?
97.63% intent-classification accuracy on a held-out 20% split of a 1,645-sample dataset covering 17 racing commands plus an out-of-scope class, with 1.39s average classifier inference on an RTX 3060 laptop GPU. This exceeds the zero-shot online models tested (best: o3-mini at 85.88%). Important caveat: the test set is synthetic LLM-generated text, not real operator speech, so real-world accuracy is unknown.
Does Jarvis run offline, and on what hardware?
Yes, entirely offline: openWakeWord for the 'Hey Jarvis!' trigger, local Whisper base.en speech-to-text, a fine-tuned Mistral-7B (4-bit QLoRA) classifier, and Coqui TTS all run locally. The final model was fine-tuned and evaluated on a consumer RTX 3060 laptop GPU; earlier benchmarking used an RTX 4090. No cloud API is required at runtime.
Is the code available?
The paper states an open-source implementation at github.com/TUM-AVS/jarvis_voice_assistant, including the fine-tuning pipeline built on Unsloth QLoRA. Explicit release of the 1,645-sample dataset and evaluation scripts is not stated in the paper text.
What are the main limitations of this work?
Four stand out: (1) the benchmark is synthetic GPT-4-augmented text, so field accuracy on real operator speech is unmeasured; (2) the reported 1.39s latency covers only the classifier, excludes speech recognition and synthesis, and was measured on different GPUs across experiment stages; (3) results are single runs without variance or significance testing; (4) no comparison against compact encoder classifiers (BERT-style), which are the natural alternative for a fixed 17-command vocabulary.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Figures 1-3 appear only as garbled inline text in the extraction; diagram content was inferred from captions and body text.
- The test-set size of ~329 samples is inferred from the stated 80/20 split of 1,645 samples, not printed explicitly; per-class details may exist in unavailable supplementary material.
- The extraction appears complete (abstract through references), but the ITSC 2026 camera-ready could differ from this arXiv v1.
- Figure 3 text contains a typo ('Hey Jarivs!') likely from extraction or the source figure; not treated as substantive.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.