Investigating the Ability of Large Language Models to Analyze Recipes for Diabetes
Revathy Venkataramanan, Aditya Luthra, Venkatesan Nadimuthu, Amit Sheth
cs.CL · cs.AI
Quick verdict
9 LLMs benchmarked on judging recipe suitability for diabetes; findings plausible, but keyword-built negatives and a missing keyword baseline weaken claims.
Summary
The paper asks whether large language models can judge whether a given recipe is suitable for people with diabetes — a task the authors argue is harder than meal planning because it demands complete retrieval of dietary guidelines, conceptual understanding, and deductive application to ingredients and cooking methods. They build a 7,607-recipe benchmark: 3,807 diabetes-suitable recipes from MayoClinic, Diabetes UK, and Diabetes Food Hub, and 3,800 unsuitable recipes sampled from Recipe1M via keywords such as 'sausage', 'cookies', and 'soda'. Nine models (Mistral 7B/12B, Gemma2 2B/9B/27B, Llama3.1 8B/70B, Llama3.2 2B, ChatGPT-3.5) are evaluated under three prompts: a bare query, a prompt embedding MayoClinic/NIDDK guidelines, and a version adding worked examples. Accuracy ranges from near chance (Gemma2-2B, 0.51 on Prompt-1) to 0.85 (Llama3.1-70B, Prompt-3); Mistral-7B leads the bare query (0.84). Precision generally exceeds recall, which the authors read as deliberate caution against false 'suitable' verdicts, and they propose stability = 1 − |P − R|, correlating guideline-keyword usage in reasoning with F1 and stability. Main limitations: the keyword-built negative class is noisy ('baked', 'pickles') and lacks a keyword-classifier baseline; there is no significance testing or repeated runs; the stability metric is ad hoc; and model versioning is inconsistent (Llama3.2 '2B' vs '3B').
Review complete
Assessment at a glance
5
risk flags identified
● high 2
● medium 3
● low 0
4 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- Most LLMs are systematically cautious, predicting 'not suitable' more readily than 'suitable' (precision exceeds recall across most model-prompt pairs in Table 1).
- Models whose reasoning draws on dietary-guideline keywords achieve higher F1 and precision-recall stability (Figure 3).
- Mistral-7B and Llama3.1-70B outperform the other tested models overall (best accuracies of 0.84 on Prompt-1 and 0.83/0.85 on Prompts 2/3, respectively).
- Providing guidelines in the prompt generally beats relying on internal retrieval, but for ChatGPT-3.5 the external guidelines conflicted with internal knowledge (Prompt-2 recall 0.09).
Strengths
Clean prompt hierarchy isolating knowledge sources
Prompt-1/2/3 factor out internal retrieval, externally supplied guidelines (MayoClinic/NIDDK), and worked examples, enabling the finding that external context helps most models (e.g., Llama3.1-70B accuracy 0.69 → 0.83 → 0.85) yet destabilized ChatGPT-3.5 (Prompt-2 recall 0.09).
Balanced, sizable benchmark with public artifacts
7,607 recipes (3,807 suitable / 3,800 not) roughly class-balanced; code, dataset, and per-model results released on GitHub (footnotes 1-2), enabling re-analysis.
Behavioral reading of precision-recall asymmetry
Novel reasoning-keyword probe
Weaknesses
Benchmark label validity is weak by construction
Missing obvious baseline
No statistical rigor
Ad hoc stability metric and partially tautological correlation
Versioning and reporting inconsistencies
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
This preprint (AI Institute, University of South Carolina) evaluates nine LLMs on binary classification of recipe suitability for diabetes, framed as three sub-challenges: medical knowledge retrieval from parametric memory, conceptual understanding of guideline categories, and deductive application to ingredients and cooking methods (§1). Central claims: (1) most LLMs are cautious, under-predicting 'suitable' (precision > recall in Table 1); (2) models whose reasoning invokes guideline keywords perform better (Figure 3); (3) Mistral-7B and Llama3.1-70B are strongest overall. The claims are directionally supported by Table 1 but rest on weak statistics and a benchmark whose label construction is itself the weakest link.
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 4 strengths & 5 weaknesses, analyzed in detail
- 5 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
Which LLM performed best at judging whether recipes are suitable for diabetes?
In this preprint, Mistral-7B was best when models had to rely on internal knowledge (0.84 accuracy on the direct-query prompt), while Llama3.1-70B was best when dietary guidelines were supplied in the prompt (0.83 accuracy on the context-guided prompt and 0.85 on the example-rich prompt).
How was the diabetes recipe benchmark dataset built?
The authors collected 3,807 recipes labeled suitable from medical sources (MayoClinic, Diabetes UK, Diabetes Food Hub) and selected 3,800 not-suitable recipes from Recipe1M by searching titles and ingredients for keywords such as 'sausage', 'cookies', 'soda', and 'baked', for a total of 7,607 recipes.
Why do the authors say LLMs are 'cautious' about diabetes recipes?
Across most models and prompts, precision exceeded recall — for example Llama3.2-2B scored 0.95 precision but only 0.03 recall with context prompts — meaning models rarely said 'suitable', which the authors interpret as avoiding false reassurance in a health-risk setting.
Is the code and dataset for this paper publicly available?
Yes. The paper links a GitHub repository (revathyramanan/LLMs-Diabetes-Recipes) containing the code, the dataset, and per-model metric files.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Figures 1–3 (per-model metric plots, keyword heatmap, correlation graph) are referenced and described in the text but their underlying values are image-only and not extractable.
- Per-model F1/stability/keyword-count tables are hosted behind an external shortlink (tinyurl.com/ycxf48bz) referenced in §4 and are not part of the provided text.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.