TriCalRAG: A Three-Strategy, Retrieval-Augmented Benchmark for On-Premise LLM-Based Root Cause Analysis in AIOps
Rohit Patel, Susil Kumar Mohanty, Jeenal Chaudhary
cs.DC · cs.AI · cs.CR · cs.ET · cs.LG
Quick verdict
RAG stabilizes zero-shot-degenerate LLM log-anomaly detection on one GPU; good calibration analysis but small scale and same-pool retrieval risk inflation.
Summary
TriCalRAG benchmarks whether open-weight LLMs served on a single workstation GPU (RTX PRO 6000, 96GB) can perform root cause analysis on system logs, comparing two models (Qwen2.5-14B, Mistral-Small-22B) under three prompting strategies (zero-shot, few-shot, RAG retrieving the top-3 similar labeled incidents via FAISS/MiniLM-L6-v2) against a DeepLog LSTM baseline across four LogHub datasets (BGL, HDFS, Thunderbird, OpenStack), 600 balanced 5-line-window incidents per run, with 3 seeds and bootstrap 95% CIs. The central finding concerns calibration, not accuracy: zero-shot prompting pushes predicted-positive rates to 0.99-1.00 on three datasets, so moderate F1 (0.67-0.72) masks near-chance accuracy (0.500-0.507), while RAG keeps 7 of 8 configurations within the authors' degenerate thresholds (versus 2 of 8 for zero-shot) and improves F1 by 0.10-0.27. Mistral-Small leads macro-F1 (0.644 vs 0.560) but degenerates in more configurations (7 vs 5 of 12) and runs at roughly half Qwen2.5-14B's throughput. Batching scales throughput 41x; AWQ 4-bit quantization cuts latency 20% with no measurable accuracy loss. Main limitations: small scale (150 incidents per dataset; DeepLog evaluated on only 120 incidents), no cloud API baseline despite the on-premise motivation, two of five planned models evaluated, RAG retrieves from the same incident pool it is evaluated on, and explanation quality is unassessed. The authors candidly disclose a parser bug and the leakage-corrected DeepLog result (0.913 to 0.698).
Review complete
Assessment at a glance
5
risk flags identified
● high 1
● medium 3
● low 1
5 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- RAG improves mean F1 by 0.10-0.27 over zero-shot prompting and keeps 7 of 8 configurations within calibrated predicted-positive-rate thresholds, versus 2 of 8 for zero-shot.
- Zero-shot prompting drives both models toward near-degenerate all-anomaly prediction (predicted-positive rates up to 1.000 on OpenStack), so moderate F1 (0.67-0.72) masks near-chance accuracy (0.500-0.507).
- Mistral-Small leads macro-F1 (0.644 vs 0.560) but degenerates in more configurations (7 vs 5 of 12) and runs at roughly half of Qwen2.5-14B's throughput (365.9 vs 713.1 tok/s).
- A single workstation GPU suffices for this task: batching scales throughput 41x (48.3 to 1991.0 tok/s from batch 1 to 128) and AWQ 4-bit quantization cuts latency 20% with no measurable accuracy loss.
- A leakage-corrected DeepLog baseline achieves F1 0.698 (P 0.584, R 0.867), between the two LLMs' macro averages, without producing natural-language explanations.
Strengths
Calibration diagnostic exposes a real F1 blind spot
Section V-C shows Mistral-Small zero-shot reaching predicted-positive rates of 0.993 (BGL), 0.987 (HDFS), 1.000 (OpenStack) with accuracy 0.500-0.507 while F1 reads 0.67-0.72 — a concrete, transferable evaluation lesson, and the RAG-vs-zero-shot contrast (7/8 vs 2/8 calibrated configs) quantifies it.
Unusual transparency about failure paths
Section VII discloses the JSON parser bug (57.1% of early Mistral-Small responses misclassified as parse failures), the DeepLog data-leakage correction (F1 0.913 down to 0.698), and the HDFS windowing mismatch; Fig 2 explicitly marks Llama-3.1-8B, 70B-AWQ, and the cloud API as planned-but-not-evaluated rather than implying coverage.
Deployment-relevant ablations
Statistical protocol above conference-floor norm
Reproducibility artifacts
Weaknesses
RAG retrieves from the same pool it is evaluated on
Small evaluation scale for benchmark claims
The motivating cloud-vs-on-premise argument is never tested
'Calibration' overstates what is measured
No inferential comparison between strategies
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
TriCalRAG (arXiv:2609.14762, IIT Jodhpur) is a benchmark paper asking a deliberately narrow question: what can one workstation GPU (RTX PRO 6000, 96GB) deliver for LLM-based root cause analysis on system logs? Its five central claims: (1) RAG improves mean F1 by 0.10-0.27 over zero-shot and stabilizes predicted-positive rates (7/8 RAG configurations within degenerate thresholds vs 2/8 zero-shot); (2) zero-shot prompting drives degenerate all-anomaly behavior (predicted-positive up to 1.000 on OpenStack) so F1 misleads — 0.67-0.72 alongside 0.500-0.507 accuracy; (3) model choice is priority-dependent — Mistral-Small macro-F1 0.644 vs Qwen2.5-14B 0.560, but 7 vs 5 degenerate configurations of 12 and roughly half the throughput (365.9 vs 713.1 tok/s, Table I); (4) batching scales throughput 41x and AWQ 4-bit cuts latency 20% with no accuracy loss; (5) leakage-corrected DeepLog (F1 0.698) sits between the LLMs. All five are supported by the presented numbers as stated; the risk concentrates in how the RAG condition is constructed.
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 5 strengths & 5 weaknesses, analyzed in detail
- 5 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What is TriCalRAG?
TriCalRAG is a benchmark testing whether open-weight LLMs running locally on a single workstation GPU can perform root cause analysis on system logs. It compares zero-shot, few-shot, and retrieval-augmented prompting across two models (Qwen2.5-14B, Mistral-Small) and four public log datasets (BGL, HDFS, Thunderbird, OpenStack), with a classical DeepLog LSTM baseline.
Does RAG actually improve LLM-based log anomaly detection?
In this paper, yes, with a caveat: retrieval-augmented prompting improved F1 by 0.10-0.27 over zero-shot, but its bigger effect was behavioral stability — 7 of 8 RAG configurations kept their predicted-anomaly rate near the true 50% base rate, versus only 2 of 8 for zero-shot, which often flagged nearly every incident as anomalous. Note the retrieval pool was the same as the evaluation pool, so some of the gain may reflect near-duplicate precedent lookup.
Which model is better for on-premise log analysis, Qwen2.5-14B or Mistral-Small?
According to the paper, it depends on priorities: Mistral-Small (22B) achieves higher macro-averaged F1 (0.644 vs 0.560) but degenerates in more prompt configurations (7 vs 5 of 12) and runs at roughly half the throughput (365.9 vs 713.1 tokens/s). Qwen2.5-14B is the safer choice when predictable behavior and speed matter more than peak accuracy.
Can you run LLM root cause analysis on a single GPU?
The authors argue yes: a 96GB RTX PRO 6000 served both models via vLLM, batching scaled throughput 41x (from 48.3 to 1991.0 tokens/s between batch sizes 1 and 128), and 4-bit AWQ quantization reduced latency 20% with no measurable accuracy loss on this task.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Figures 1-3 are available only as caption text; per-dataset F1 values beyond those quoted in prose, and bootstrap CI bound values, cannot be verified from the extraction.
- The metadata abstract is truncated mid-sentence; the full abstract embedded in the body text was used instead.
- Per-configuration tables (model x dataset x style F1 matrix) are not in the extraction, so claims about few-shot behavior on HDFS/OpenStack rest on the prose numbers only.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.