A Tool-Augmented, GPT-4 Chatbot for Real-Time Repository Data Analysis
Muhammad Jawad Chowdhury, Md. Sakib Khan
cs.AI · cs.SE
Quick verdict
Claims a prompt+tool-selection GPT-4 chatbot hits 98.48% on 66 repo queries; plausible but a tiny AI-generated eval, no baselines or ablations.
Summary
The paper presents a chatbot architecture that answers natural-language questions about GitHub repositories by combining GPT-4 with tool calls to the GitHub REST API, instead of the embedding-plus-retrieval pipelines of prior systems such as MSRBot. The pipeline parses a user query, classifies it via keyword matching into categories (issues, pull requests, commits, compound, general), applies extracted filters, selects one of two tools (a repository report tool or a commits/issues/PR report tool), fetches live data, and lets GPT-4 compose the answer; queries with no matching tool fall back to internal knowledge. Evaluation uses 80 questions — 66 in-scope, 14 out-of-scope — generated by an AI model from 15 exemplar queries of Abdellatif et al. The model reportedly answered 65 of 66 in-scope questions correctly (98.48%), with the single failure on a date-range filter, and abstained correctly on all 14 out-of-scope questions; iteration counts ranged from one (simple) to five (out-of-scope). Main limitations: no baseline or prompt ablation despite claims that engineered prompts 'improve accuracy'; the 'Impact of Tool Availability' section contains speculation rather than an experiment; the repositories used, exact prompts, and code are undisclosed; no user study, latency, or cost data supports the 'real-time' and accessibility claims; and a single run ignores LLM non-determinism.
Review complete
Assessment at a glance
5
risk flags identified
● high 2
● medium 2
● low 1
5 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- A parse-classify-select-generate architecture with only two GitHub API tools and prompt engineering answers repository questions with 98.48% accuracy on 66 in-scope questions (Sec V.D).
- GPT-4 within this setup correctly abstains on all 14 out-of-scope questions rather than fabricating answers (Sec V.A, V.D).
- Relying on prompt engineering and tool selection 'inverts' prior multi-component embedding/retriever designs and suffices without document retrievers (Abstract, Sec III).
- Precisely engineered prompts enhance model accuracy and efficiency (Sec VI) — asserted without any ablation.
Strengths
Explicit abstention testing
Sec V.A/V.D include 14 deliberately out-of-scope questions; the model declined all 14 instead of fabricating answers — a hallucination check most similar demo papers omit.
Concrete, fully specified architecture
Sec III defines the input composition I = {Msys, Msum, Quser}, a system-prompt template with repository URL and tool descriptions, the two tools, and Algorithm 1's conditional tool-vs-internal-knowledge path — enough to understand the design end to end.
Honest single-failure reporting
Grounding against live repository data
Processing-cost transparency
Weaknesses
No baselines or prompt ablation
Section V.C promises an experiment it does not contain
Undisclosed evaluation setup
Single-run statistics
Title and audience claims outrun the evidence
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
The paper presents “Github-chatbot,” a tool-augmented GPT-4 system for answering natural-language questions about GitHub repositories (arXiv 2609.07586, cs.AI/cs.SE). The stated contrast with prior systems such as MSRBot [26] and AnswerBot [27] is architectural: no embedding models or document retrievers — instead, query parsing, keyword-based classification f: Q→C (Issues / Pull Requests / Commits), extraction of user filters F (state, author, date range, limit), selection of one of two tools (Repository Report; Commits/Issues/PR Report), live retrieval via the GitHub REST API, and GPT-4 response composition (Sec III). Claimed contributions: (1) the tool-augmented architecture; (2) an 80-question evaluation dataset with out-of-scope abstention items; (3) headline results — 65/66 in-scope correct (98.48%) and 14/14 correct abstentions; (4) the conclusion's assertion that “precise prompts enhance model accuracy and efficiency” (Sec VI), which no experiment in the paper tests.
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 5 strengths & 5 weaknesses, analyzed in detail
- 5 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What accuracy does the GPT-4 GitHub repository chatbot achieve?
The authors report 65 of 66 in-scope questions answered correctly (98.48%), plus correct refusal on all 14 out-of-scope questions, on a custom set of 80 questions that were AI-generated from 15 exemplar queries. The single error involved applying a date-range filter; there was only one evaluation run, so this figure should be read as a small-sample point estimate.
How does this chatbot differ from RAG-based repository assistants?
Instead of embedding models plus document retrievers, it parses the user query, classifies it by keyword matching, extracts filters, picks one of two GitHub API tools (a repository report tool and a commits/issues/PR report tool), fetches live repository data, and lets GPT-4 compose the answer. Queries with no matching tool fall back to the model's internal knowledge. Notably, the paper claims this design is better but never actually compares it against a retrieval-based baseline.
Is the code, dataset, or prompt set available for this paper?
No. The paper does not state any code, dataset, or full-prompt release. The evaluated repositories are not named, the exact prompts are shown only as a template, and the 80-question dataset is not published, which makes independent reproduction currently impossible.
Was the system tested with real users or measured for latency?
No. Despite the title's 'Real-Time' and the abstract's claims about accessibility for non-technical stakeholders, the paper reports no user study, no latency measurements, and no cost analysis — only answer accuracy, abstention behavior, and iteration counts.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Figures 1-3 appear in the extraction only as captions and axis labels (e.g., the numeric iteration-frequency values of Fig 3 are not readable); all quantitative results cited in this review come from the body text, which is complete through the references.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.