Utkarsh Soni

—

Unclaimed · AI-structured profileOpenAlex

NLP researcher probing LLM agent reliability: long-horizon benchmarks, synthetic-data forgetting, prompt optimization.

papers here

reviewed by Evolve

first-author

corresponding

—

works · OpenAlex

h-index · OpenAlex

Bibliometric figures come from OpenAlex (2026-10-17) and cover the author's entire public record, not just the papers listed here. Reproducibility signal comes from Evolve's own reviews.

Research profile

Output—

Output scoring is not yet available: the pipeline lacks a complete record of this researcher's total publications, venues, and output volume. This axis would measure annual publication counts, venue quality, and the consistency of scholarly production over time.

Impact—

Impact scoring is not yet available: citation aggregates beyond basic h-index inputs are insufficient for a reliable estimate. This axis would measure citation footprint, influence on follow-up work, and recognition within the LLM evaluation and agents communities.

Focus0/100

Focus scores 0, the flattest profile in the record: topics are spread almost evenly — Data Visualization and Analytics (8), Topic Modeling (6), AI-based Problem Solving and Planning (6), Artificial Intelligence in Games (5), XAI (5) — and the current field split is exactly 50/50 between LLM agents and LLM evaluation and benchmarks. Earlier visualization-era work and the present LLM agenda coexist, so no single theme yet dominates.

Collaboration44/100

Collaboration scores 44, a moderate figure. The OpenAlex record lists 8 distinct coauthors, and the frequent-coauthor set — Syed Shariyar Murtaza, Yifan Nie, Eugene Wen, Wentao Zhang, Junaid Ahmad Bhatti, Yuntian Deng, Arvid Frydenlund, Sachin Chandrasekhar — suggests a small recurring circle rather than a large network. Authorship positions (first on the TAM benchmark, middle on the two agent/systems papers) show both leading and supporting collaboration roles.

Activity52/100

Activity scores 52. The counts-by-year record spans 2016 to 2026 (2016: 1, 2018: 3, 2019: 1, 2020: 3, 2021: 3, 2022: 2, 2023: 2, 2025: 2, 2026: 3) with gaps in 2017 and 2024, and recent3yWorks stands at 5. The three 2026 arXiv releases form the densest burst in the record, though the overall cadence remains moderate.

Reproducibility55/100

Reproducibility scores 55, computed as a mean (reproMean 55) over 2 papers with readiness assessments (reproPapers 2). Within the Evolve catalog, Control-Data Flow Separation is rated high readiness, while the TAM benchmark paper and the synthetic-data forgetting study are both medium — benchmark and fine-tuning studies that would benefit from released code, label-quality analysis, and fuller protocol documentation.

Research summary

Utkarsh Soni is a natural language processing researcher whose recent work centers on making large language model systems measurable and dependable. The OpenAlex record splits the field distribution evenly (0.5 and 0.5) between LLM agents and LLM evaluation and benchmarks, and topic records echo this breadth: Data Visualization and Analytics (8), Topic Modeling (6), AI-based Problem Solving and Planning (6), Artificial Intelligence in Games (5), and Explainable AI (5). Three 2026 arXiv papers define the current agenda. As first author, Soni introduced the TAM benchmark on application-manual tasks, showing GPT-5 near-fails long-horizon procedures with 1%–15.5% exact match; Evolve flags it as valuable though single-model with an unquantified label-noise ceiling. A middle-author study shows synthetic-data LoRA fine-tuning causes catastrophic forgetting in LLM-agent skill retrieval, with low adaptation rank outweighing regularizers. A third paper proposes control-data flow separation for stable prompt optimization in multi-agent LLMs, keeping protocols 100% valid by construction. Indicators show h-index 7, i10-index 6, 8 distinct coauthors, and 5 works in the last three years, with output recorded from 2016 through 2026. Citation-impact and total-output scores remain unavailable in the current data.

Collaboration pattern

Utkarsh Soni works within a compact, recurring circle: the OpenAlex record lists 8 distinct coauthors, led by frequent collaborators Syed Shariyar Murtaza, Yifan Nie, Eugene Wen, Wentao Zhang, Junaid Ahmad Bhatti, Yuntian Deng, Arvid Frydenlund, and Sachin Chandrasekhar. The 2026 pattern mixes roles — first-authoring the TAM benchmark while holding middle-author positions on the synthetic-data and prompt-optimization papers — consistent with a collaboration score of 44 (moderate network breadth).

Research keywords

LLM evaluationLLM agentsbenchmark constructionlong-horizon procedural reasoningretrieval-augmented generationprompt optimizationmulti-agent LLM systemscatastrophic forgettingcontinual learningsynthetic dataskill retrievalexplainable AI

Highlights

TAM benchmark shows GPT-5 near-fails long application-manual procedures
As first author, introduced Tasks over Application Manuals, revealing GPT-5 exact-match scores of only 1%–15.5% on long-horizon procedural reasoning tasks spanning ICD-10-CM coding and federal sentencing guidelines; Evolve values the benchmark but notes single-model scope and an unquantified label-noise ceiling.
Synthetic-data LoRA tuning triggers forgetting in LLM-agent skill retrieval
A middle-author study documents catastrophic forgetting when the skill retrieval of LLM agents is LoRA-tuned on synthetic data; regularized tuning preserves out-of-distribution behavior, but low adaptation rank outweighs the regularizers.
Control-data flow separation stabilizes multi-agent prompt optimization
Proposes freezing control channels during prompt optimization (e.g., TextGrad) in multi-agent LLM systems, keeping structured-output protocols 100% valid by construction; Evolve notes the quality gains are modest and untested.

Timeline

2026First-author arXiv release of Tasks over Application Manuals, a benchmark exposing GPT-5's 1%–15.5% exact match on long-horizon application-manual procedures.
2026Co-authored study on catastrophic forgetting in skill retrieval for LLM agents under synthetic-data LoRA fine-tuning.
2026Co-authored Control-Data Flow Separation, a prompt-optimization stability method for multi-agent LLMs rated high in reproduction readiness.

Papers in the Evolve catalog

2026-09-11first author
Tasks over Application Manuals: Revealing Gaps in Long-Horizon Procedural Reasoning for Language Models
2026-09-09
When Synthetic Data Hurts: On Catastrophic Forgetting in Skill Retrieval for LLM Agents
2026-09-01
Control-Data Flow Separation: Stable Prompt Optimization in Multi-Agent LLMs

Frequent co-authors

Syed Shariyar Murtaza×3 Yifan Nie×3 Eugene Wen×3 Wentao Zhang×1 Junaid Ahmad Bhatti×1 Yuntian Deng×1 Arvid Frydenlund×1 Sachin Chandrasekhar×1

About this profile

This profile is AI-structured by Evolve.Pub from public bibliographic data (OpenAlex) and Evolve's own paper reviews. It is not written or endorsed by the researcher unless the profile is claimed. Radar dimensions are computed from real bibliometrics with fixed formulas — raw figures are shown alongside; missing dimensions mean insufficient data, never zero. Errors? Claim the profile to correct or hide it.

Bibliometric indicators (h-index 7, i10-index 6, yearly counts, topics) come from OpenAlex only; identity is name-resolved from public bibliographic data.
No institution, country, or ORCID is on record, so no affiliation or geographic claims are made here.
Citation impact and total output are unscored due to insufficient data; the reproducibility score rests on only 2 assessed papers.
All three catalog papers are 2026 arXiv preprints; peer-reviewed venue status is not reflected in this profile.