Umesh Bodhwani

Research profile

Research summary

Umesh Bodhwani is an early-career NLP researcher at Amazon whose reviewed work centers on making large language model systems more measurable and more trustworthy. The Evolve catalog lists three 2026 arXiv preprints, each with Bodhwani as first author. GAUGE examines when LLM-as-a-judge panels can be trusted in user-simulated evaluation of task-oriented agents, finding that satisfaction signals mislead release gates while rankings hold; Evolve calls it a credible audit with strong controls, limited by a small judge panel and proprietary models. A second preprint proposes calibrated reflection prompting with ordinal classification for LLM confidence estimation, where discrimination gains appear real but headline calibration metrics lose to a log-probability baseline. The third, LentEx, uses LoRA-tuned Mistral-7B trained on synthetic data for latent entity extraction, reporting clustering and retrieval-augmented generation gains despite circular evaluation and no significance tests. OpenAlex bibliometrics confirm the early stage: h-index 1, i10-index 0, and 3 works in the last three years, with OpenAlex counting 2 works in 2025 and 1 in 2026. Topic spread covers five OpenAlex topics led by Topic Modeling (2), and the field split is even at roughly one third each across LLM agent evaluation, NLP, and LLM confidence estimation — a broad, exploratory agenda rather than a narrow specialization. Collaboration is mid-range (46/100) with 9 distinct coauthors.

Research keywords

Highlights

• GAUGE: auditing when LLM judges can be trusted in agent evaluation

  First-authored 2026 arXiv audit of LLM-as-a-judge in user-simulated evaluation of task-oriented agents. Finding: satisfaction signals mislead LLM-judge release gates while rankings hold. Evolve calls it a credible audit with strong controls, limited by a small judge panel and proprietary models; reproducibility readiness is medium.

• Calibrated reflection prompting for LLM confidence estimation

  First-authored 2026 arXiv paper proposing reflection prompting plus ordinal classification to improve verbalized confidence estimation. Evolve judges the discrimination gains real, but headline calibration metrics (expected calibration error) lose to a log-probability baseline; reproducibility readiness is low.

• LentEx: latent entity extraction via synthetic data and LoRA tuning

  First-authored 2026 arXiv paper training a LoRA-tuned Mistral-7B on synthetic data for generalizable latent entity extraction. Evolve notes real clustering and retrieval-augmented generation gains, but flags circular evaluation and missing significance tests; reproducibility readiness is medium.

Timeline

1. 2025OpenAlex countsByYear records 2 works in 2025, the earliest activity in the current snapshot; these works are not among the three reviewed preprints.
2. 2026GAUGE preprint posted on arXiv: a first-author audit of LLM-as-a-judge reliability in user-simulated evaluation of task-oriented agents.
3. 2026A Calibrated Reflection Approach for Enhancing Confidence Estimation in LLMs posted on arXiv, another first-author paper combining reflection prompting with ordinal calibration.
4. 2026LentEx preprint posted on arXiv: first-author work on latent entity extraction using synthetic data and a LoRA-tuned Mistral-7B.

Papers in the Evolve catalog

• 2026-09-10first author

  GAUGE: When Not to Trust LLM-as-a-Judge in User-Simulated Evaluation of Task-Oriented Agents

• 2026-09-03first author

  A Calibrated Reflection Approach for Enhancing Confidence Estimation in LLMs

• 2026-09-03first author

  LentEx: Generalizable Latent Entity Extraction via Synthetic Data and Instruction-Tuned LLMs

Frequent co-authors