Can LLMs Extract Architectural Design Decisions from Source Code Commits? - A Preliminary Exploratory Study
Amey Karan, Rudra Dhar, Mohamed Soliman, Karthik Vaidhyanathan
cs.SE · cs.AI
Quick verdict
Preliminary n=30 study: LLMs recover ADDs from commits moderately (BERT-F1>0.81) but miss rationale; candid work, though small sample, no significance tests.
Summary
This preliminary exploratory study asks whether large language models can extract Architectural Design Decisions (ADDs) directly from source code commits. The authors curate 30 developer-written ADDs (60-100 words each, mapped to a single commit) from the Maestro dataset, spanning Apache Cassandra, Hadoop, and Tajo, and prompt four LLMs - Gemini 3 Pro, DeepSeek R1, Kimi K2 Thinking, and Qwen3 32B - with the commit message and git diff under zeroshot and embedding-based dynamic fewshot prompting. Outputs are scored with ROUGE-L, BLEU, METEOR, and BERTScore, and one author manually reviews all 60 Gemini outputs against an a priori rubric covering abstraction level, verbosity, and rationale. All models exceed 0.81 BERT-F1; Gemini scores best (0.828 zeroshot, 0.847 fewshot) although its METEOR drops from 0.175 to 0.165. Lexical overlap stays low (ROUGE-L at most 0.152, BLEU at most 0.014), indicating paraphrase rather than copying. Manual inspection finds outputs 2-10x longer than the documented ADDs, heavy with implementation details such as class names, and usually missing decision rationale; fewshot prompting fixes length and style but not rationale. Limitations include the small, selectively curated, Java-biased dataset, single-run generation without variance reporting or significance tests, a single-reviewer inspection limited to Gemini, and unquantified pretraining-leakage risk from public Apache data. The authors conclude that ADD extraction is architectural reasoning rather than summarization and propose augmenting commits with issue discussions and reviews.
Review complete
Assessment at a glance
5
risk flags identified
● high 0
● medium 3
● low 2
5 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- All four tested LLMs (Gemini 3 Pro, DeepSeek R1, Kimi K2 Thinking, Qwen3 32B) recover ADDs from commit message plus git diff with BERT-F1 above 0.81 under zeroshot prompting (Table 2).
- Dynamic embedding-based fewshot prompting improves alignment for all four models on most metrics, e.g., Gemini BERT-F1 0.828 to 0.847, with Gemini's METEOR as the reported exception (0.175 to 0.165).
- Manual inspection of all 60 Gemini outputs shows generated ADDs are typically 2-10x longer than documented ones, contain code-level details, and omit decision rationale.
- Fewshot examples teach expected length and style but not rationale, because commits encode what changed rather than why, making ADD extraction an architectural reasoning task rather than summarization.
Strengths
Addresses a genuine gap in ADD recovery
Sections 1-2 position the work against prior extraction approaches (Bhat et al., ADeX, Shahbazian et al.) that depend on rich textual artifacts like issue trackers and mailing lists; evaluating LLMs on commit message + git diff alone is the underexplored configuration the paper targets.
Mixed-method evaluation with an a priori rubric
Section 3.3 fixes three rubric dimensions (abstraction level, verbosity, rationale) and criteria before inspection and applies them uniformly to all 60 Gemini outputs; Section 4 grounds the findings in concrete cases (CASSANDRA-15505, YARN-5889, CASSANDRA-3411) rather than aggregate scores alone.
Replication package provided
Candid reporting of negative and awkward results
Transparent, honest threats-to-validity section
Weaknesses
No statistical treatment of the quantitative results
No baselines calibrate the headline metric
Manual inspection is single-reviewer and single-model
Leakage risk acknowledged but untested
Fewshot retrieval configuration underspecified in-text
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Preliminary Review: Can LLMs Extract Architectural Design Decisions from Source Code Commits?
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 5 strengths & 5 weaknesses, analyzed in detail
- 5 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
Can LLMs extract architectural design decisions from code commits?
According to this preliminary study, moderately well: four LLMs (Gemini 3 Pro, DeepSeek R1, Kimi K2 Thinking, Qwen3 32B) prompted with a commit message and git diff reached BERT-F1 above 0.81 against developer-written decisions, but their outputs were too long, full of implementation details, and usually missed the rationale behind the decision.
Which LLM performed best for extracting design decisions from commits?
Gemini 3 Pro scored highest: BERT-F1 0.828 zero-shot and 0.847 with dynamic few-shot examples, ahead of Qwen3 32B (0.823 and 0.829). Notably, Gemini's METEOR dropped from 0.175 to 0.165 when few-shot examples were added.
Why do LLM-generated design decisions miss the rationale?
The authors argue commits record what changed, not why: the motivating reasoning usually lives outside the commit, in issue discussions, design documents, or code reviews, so a model given only the commit message and diff has little basis to recover the 'why'.
Is the dataset and code for this study available?
Yes. The paper provides a replication package containing the 30-ADD dataset, prompts, and scripts on Zenodo, DOI 10.5281/zenodo.20996340.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.