NLP evaluation researcher at Carnegie Mellon University building pre-registered audits of LLM fairness and robustness
3
papers here
3
reviewed by Evolve
2
first-author
0
corresponding
—
works · OpenAlex
1
h-index · OpenAlex
Bibliometric figures come from OpenAlex (2026-10-16) and cover the author's entire public record, not just the papers listed here. Reproducibility signal comes from Evolve's own reviews.
Research profile
Output—
Output metrics are not yet computed for this profile: the snapshot lacks reliable totals for works, venues, and productivity volume, so this axis — which would measure publication volume and output consistency over time — is left unscored rather than estimated.
Impact—
Impact cannot be scored yet: the snapshot shows only an h-index of 1 and an i10-index of 0, far too little citation signal for a meaningful score; this axis would measure citation uptake, influence on follow-up work, and downstream reuse.
Focus0/100
The focus score of 0 reflects extreme breadth rather than missing data: the field distribution splits exactly evenly at one-third each across NLP/AI fairness evaluation, retrieval-augmented generation evaluation, and LLM evaluation, while topic assignments cluster into five areas — topic modeling, NLP techniques, domain adaptation and few-shot learning, explainable AI, and ML classification — with two mentions apiece. No single direction yet dominates the profile.
Collaboration32/100
The collaboration score of 32 sits low: the recent window lists only 4 distinct co-authors, and the profile names four frequent collaborators — Manikandan Ravikiran, Runmin Jiang, Xiaomo Li, and Min Xu. Authorship patterns reinforce a compact team structure: first author on the fairness audit and the RAG identity-handoff audit, last author on the multiple-choice robustness audit. The network is small but recurs across evaluation projects.
Activity47/100
Activity scores 47, an intermediate reading on thin data: the recent three-year window contains 4 works, and OpenAlex counts-by-year shows single works in 2012, 2020, and 2022 before a step up to 4 works in 2026 — the year of all three Evolve-reviewed papers. The pattern suggests a burst of evaluation research concentrated in 2026 rather than sustained high-volume production.
Reproducibility85/100
Reproducibility is the profile's strongest axis at 85, the mean across 3 audited papers. Two of the three Evolve-reviewed papers carry a high reproducibility rating — the pre-registered fairness audit using equivalence testing, and the RAG identity-handoff audit described as careful and reproducible — while the NC-MCAR robustness audit rates medium, dinged for a missing noise-floor baseline. Methodological transparency is a consistent signature.
Research summary
Siddharth Vohra is a researcher at Carnegie Mellon University (US) whose work centers on evaluation methodology for large language models — auditing how model decisions are measured rather than only how models perform. Identity is resolved from OpenAlex bibliographic data. The Evolve catalog reviews three 2026 arXiv papers: a pre-registered, 40,000-request correspondence audit finding no LLM demographic bias surviving correction in any format (first author); a robustness audit introducing the NC-MCAR metric, showing four large models flip originally correct answers after spurious expert prompts (41.1% versus 12.5% under majority prompts; last author); and an audit of identity handoffs in retrieval-augmented generation pipelines, finding ranked retrieval can silently drop the selected object (first author, with reader-effect evidence based on a curated 64-item cohort). Bibliometric signals are early-stage: h-index 1, i10-index 0, four works in the recent three-year window, four distinct co-authors, and counts-by-year entries in 2012, 2020, 2022, and 2026. What distinguishes the profile is methodological rigor: Evolve rates reproducibility at 85, the mean across three papers, with two of three graded high readiness — reflecting pre-registration, paired audit designs, and equivalence testing. Topic coverage is deliberately broad — fairness evaluation, LLM evaluation, and RAG evaluation split evenly at one-third each — yielding a focus score of 0, while output and impact scores are unavailable in the current snapshot.
Collaboration pattern
The snapshot lists 4 distinct co-authors, with Manikandan Ravikiran, Runmin Jiang, Xiaomo Li, and Min Xu as frequent collaborators. Authorship positions reinforce a compact team: first author on two of the three reviewed audits and last author on the third, a pattern consistent with leading evaluation studies while also contributing as senior author. The collaboration dimension scores 32, indicating a small network of repeat collaborators rather than an expansive co-author web.
Research keywords
Highlights
Pre-registered 40k-request audit finds no surviving demographic bias
Models flip correct answers under spurious expert cues
Ranked retrieval silently drops the selected object
Timeline
- 2012Earliest record in the OpenAlex counts-by-year: 1 work attributed to this identity.
- 2020A single work appears in the OpenAlex counts-by-year, a second sparse data point.
- 2022One further work is recorded before the 2026 concentration.
- 2026Most active year: 4 works in counts-by-year, including all three Evolve-reviewed papers — the 40k-request fairness audit, the NC-MCAR robustness audit, and the RAG identity-handoff audit.
Papers in the Evolve catalog
- first author
The Audit Decides the Verdict: Instrument Effects Rival Demographic Bias in LLM Decision Audits
When Models Defer to Wrong Answers: A Robustness Audit of Source-Attributed Cues in Multiple-Choice QA
- first author
Does the Selected Object Reach the Reader? Auditing Identity Handoffs in Grounded Language-Model Pipelines
Frequent co-authors
About this profile
This profile is AI-structured by Evolve.Pub from public bibliographic data (OpenAlex) and Evolve's own paper reviews. It is not written or endorsed by the researcher unless the profile is claimed. Radar dimensions are computed from real bibliometrics with fixed formulas — raw figures are shown alongside; missing dimensions mean insufficient data, never zero. Errors? Claim the profile to correct or hide it.
- Bibliometric indicators (h-index 1, i10-index 0, counts-by-year) come from an OpenAlex snapshot and may undercount citations and works.
- No ORCID is linked; the profile is name-resolved from OpenAlex public bibliographic data, and 'Siddharth Vohra' may match more than one researcher.
- Output and impact dimension scores are unavailable in the source data (dimMissing) and were not estimated.
- Qualitative assessments rest on the three Evolve-reviewed arXiv papers; no journal or conference venue information was provided.
