Autonomous Research for Open-Ended Problems: A Case Study on Telecom Ticket Retrieval
Junghyun Min, Huseyin Uzunalioglu, Mohamed Trabelsi
cs.AI · cs.IR · cs.LG
Quick verdict
Case study: LLM agents reach 90% of human SoTA (0.34 vs 0.38 R@1) on telecom ticket retrieval; candid but single-run comparisons lack statistics.
Summary
The paper asks whether fully autonomous LLM-agent research pipelines—successful so far on narrow benchmarks—can handle an open-ended, industry-grade ML task. In a case study, the authors (a Georgetown intern and Nokia Bell Labs researchers) run eight autonomous research campaigns on proprietary telecom ticket retrieval (TTR): ~250k tickets plus 204k fault analyses and 89k technical analyses, with 7.6k held-out queries evaluated by Recall@k. Campaigns cross three design axes—single- vs multi-agent frameworks (Karpathy's autoresearch vs AutoScientists), agent choice (Claude Sonnet 5, Cursor Composer 2.5, GPT-OSS 120B), and informedness (human SoTA techniques disclosed in documentation)—under an operational harness of documentation, explicit search spaces, and a scripted bash loop. The best discovered system, a single fine-tuned MPNet with 2× T-T pair oversampling and metadata-based hard negatives, reaches Recall@1 of 0.343 versus 0.380 for an unpublished internal SoTA—about 90%—while beating BM25 (0.072), fine-tuned MPNet (0.200), and the TeleDoCTR ensemble (0.251); it cost ~$150-200 on Cursor and took 10 weeks versus ~10 months of human engineering. Surprisingly, framework, agent, and informedness barely affected peak performance, and no agent implemented re-ranking, augmentation, or ensembling even when told. Main limitations: one proprietary task, uncontrolled single-run comparisons, and confounded design axes, which the authors acknowledge candidly.
Review complete
Assessment at a glance
5
risk flags identified
● high 0
● medium 3
● low 2
5 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- Autonomous LLM-agent research reaches 90% of internal human SoTA (R@1 0.343 vs 0.380) on telecom ticket retrieval in 10 weeks at roughly $150-200 per Cursor campaign.
- Framework structure (single vs multi-agent), underlying LLM agent (Claude Sonnet 5, Cursor Composer 2.5, GPT-OSS 120B), and informedness had negligible effect on peak discovered performance.
- Agents excel at narrow hyperparameter and representation tuning but never implement re-ranking, data augmentation, or model ensembling even when these techniques are explicitly documented for them.
- Autonomous campaigns require an operational harness and human babysitting: agents underuse GPUs, stall, duplicate experiments, or fail at orchestration (GPT-OSS spawned sub-agents once in dozens of attempts).
Strengths
Transparent null results rarely reported in this space
Section 4.3 and Table 3 document that framework structure, agent choice, and informedness had negligible effect on peak R@1, and that no agent implemented re-ranking, data augmentation, or ensembling even when explicitly documented (footnote 7 concedes the informed agent never used the information) — valuable negative evidence for autonomous-ML research.
Reusable operational harness with concrete failure catalogue
Section 3.2 replaces unreliable agentic self-looping with a deterministic bash resume loop, assigns explicit search spaces per team/variable, and reports quantified failure modes: GPT-OSS 120B spawned sub-agents once in dozens of attempts, Qwen Coder 30B failed outright, Cursor duplicated experiments, Claude debugged environment scripts.
Fully specified winning recipe
Realistic industrial testbed against multiple human baselines
Unusually candid Limitations section
Weaknesses
No statistical treatment; one run per configuration
Headline 90%-of-SoTA framing is metric-selective
Efficiency claim (10 weeks vs 10 months) is not controlled
Proprietary data, unpublished comparator, no availability statement
Informedness null may be a manipulation artifact
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
The paper (arXiv 2609.13073; a Georgetown internship work at Nokia Bell Labs) asks whether end-to-end autonomous LLM-agent research — demonstrated so far mainly on narrow benchmarks like language-modeling loss — transfers to an open-ended, industry-grade task: telecom ticket retrieval (TTR). Three research questions structure it: performance versus human research (RQ1), time and API cost (RQ2), and empirical advantages/limitations (RQ3). Central claims: (1) the best agent-discovered system attains Recall@1 0.343 versus an internal human SoTA of 0.380 (the abstract's 90%), beating BM25 (0.072), fine-tuned MPNet (0.200), and the TeleDoCTR ensemble (0.251) — Table 2; (2) campaigns cost ~$150-200 each on Cursor (2-3x on Claude; ~free on local GPT-OSS) and 1 GPU-week over 10 weeks, versus ~10 months of prior human engineering; (3) framework structure, agent choice, and informedness negligibly affect peak performance (Table 3), and no agent implemented re-ranking, augmentation, or ensembling even when these were documented for it.
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 5 strengths & 5 weaknesses, analyzed in detail
- 5 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What performance did autonomous AI research achieve on telecom ticket retrieval?
The best agent-discovered system — a single fine-tuned MPNet retriever with 2× oversampling of peer-ticket pairs and metadata-based hard negatives — reached Recall@1 of 0.343, about 90% of the human-designed internal state-of-the-art (0.380), while clearly beating BM25 (0.072), a standard fine-tuned MPNet (0.200), and a human-designed ensemble (0.251).
How much does it cost to run an autonomous ML research campaign with LLM agents?
In this case study, a 10-20 experiment campaign cost an estimated $150-200 in API charges with Cursor Composer 2.5, roughly 2-3x more ($300-600) with Claude Sonnet 5, and essentially nothing for locally hosted GPT-OSS 120B — plus about 1 GPU-week of compute and 10 weeks of wall-clock time.
Can LLM research agents replace human ML researchers?
Not according to this paper. Agents were strong at narrow hyperparameter optimization but never proposed re-ranking, data augmentation, or ensembling even when told about them, needed operational babysitting, and the authors recommend human-AI collaboration rather than replacement.
Did the choice of LLM agent matter for the autonomous research results?
Surprisingly little: single-agent campaigns with Claude Sonnet 5, Cursor Composer 2.5, and GPT-OSS 120B all converged to Recall@1 between roughly 0.30 and 0.34, and the locally hosted open-weight GPT-OSS performed comparably to the commercial agents.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.