Building a research-software catalog with a coding agent: from hackathon prototype to public deployment
Kazuyoshi Yoshimi, Satoshi Terasaki, Gotai Yamada
cs.SE · cs.AI · cs.CY · physics.ed-ph
Quick verdict
Experience report claiming coding agents built a catalog fast but silent-failure hardening dominated; credible, honest, yet single-team and code-unreleased.
Summary
This experience report from the University of Tokyo's Institute for Solid State Physics documents two systems. The first is a research-software catalog built with the Claude Code coding agent during a three-day hackathon (68 repositories) and then hardened for public deployment: a four-stage pipeline with JSON interfaces, readiness scoring on seven binary signals, and daily GitHub Actions runs. Eight days and 54 commits resolved 25 issues, many surfaced by adversarial review from a second agent (Codex); the most consequential were silent failures — rate-limited README fetches misrecorded as missing, privacy leakage through shared record objects, stale build-time timestamps. The second is an exploratory retrieval agent for the human-curated MateriApps portal (336 applications, 49,485 indexed passages) combining curated-metadata routing, multilingual vector search, BGE-M3 cross-encoder re-ranking, and local LLM generation. On auto-generated keyword questions (34 Japanese, 33 English), dictionary routing is near-perfect on template wording but no better than unrestricted search on paraphrases; HNSW at default ef=100 loses recall as the index scales (Japanese hit@5 0.914→0.949 at ef=500); and an over-aggressive deduplication rule silently deleted one package's 22 installation pages. The authors conclude that AI-assisted portals require explicit validation and monitoring, with curated metadata remaining essential. Limitations: no controlled comparison with conventional development, auto-generated evaluation questions, and source code not publicly released.
Review complete
Assessment at a glance
4
risk flags identified
● high 0
● medium 3
● low 1
4 strengths · 4 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- A coding agent enabled a working 68-repository catalog within a three-day hackathon, but reliable public deployment required most of the engineering effort, organized around 25 issues over eight days.
- The most consequential failures are silent — plausible but incomplete or incorrect outputs — catchable only through explicit validation, monitoring, and repeated adversarial review.
- Curated metadata and maintained documentation remain essential: dictionary routing fires only on verbatim application names, and 62 of 336 MateriApps applications had no retrievable external documentation.
- Scaling retrieval from 68 records to a 49,485-passage corpus exposed failure modes in routing, HNSW approximate search, corpus construction, and aggregate evaluation that the small system did not exhibit.
Strengths
Credible silent-failure taxonomy with measured examples
§3.2.1 documents rate-limited README fetches scored as missing READMEs; §3.2.2 the shared-record-object slug rewrite that could alter a public URL after the leak check; §4.3 the dedup rule that erased one package's 22 installation pages (17 vs 183 passages after correction). Each is paired with an operational countermeasure, e.g. strict runs fail when ≥5 requests are attempted and >20% fail.
Negative results for retrieval components reported honestly
Table 1 shows dictionary routing adds nothing on paraphrased questions (JA 0.647/0.332, identical to unrestricted search) and discloses the pre-fix English failure (0.515/0.252; all seven activations false positives). §4.1 also reports that the hierarchical two-stage variant restricts candidates too early.
Unusually strong data availability
Transferable operational principles and calibrated claims
Weaknesses
Effort claims rest on anecdote
Source code and evaluation question sets not public
No uncertainty on retrieval metrics; small auto-generated sets
Readiness indicators validated only by inspection
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
This is an experience report from the Institute for Solid State Physics (U. Tokyo) covering two artifacts: (1) a research-software catalog built with the Claude Code coding agent during the three-day DxMT AIMHack 2026 hackathon (68 repository records), subsequently hardened for public deployment; (2) an exploratory retrieval agent for the human-curated MateriApps portal (336 applications; 49,485 indexed passages; crawl completed 3 August 2026). The stated contributions (§1) are documenting the prototype-to-deployable transition, identifying a class of silent failures distinct from LLM hallucination, and illustrating how curated metadata, automated collection, and retrieval complement each other. The paper is disciplined about scope: every quantity is snapshot-dated (catalog: 1 September 2026; portal listing: 31 July 2026), and the entire MateriApps half is explicitly labeled preliminary (§4, §5.4, §6).
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 4 strengths & 4 weaknesses, analyzed in detail
- 4 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What does this paper say about building software with AI coding agents?
The authors built a 68-repository research-software catalog during a three-day hackathon using the Claude Code agent, then spent roughly eight more days and 54 commits fixing 25 issues before public deployment. Their key finding: the worst problems were 'silent failures' — the system kept running but produced plausible-looking wrong results, such as recording a rate-limited download as 'no README' — which only explicit validation and monitoring caught.
What is a 'silent failure' in this paper?
A failure mode where software keeps operating and emits plausible output instead of crashing. Examples include: a failed README download scored as a missing README, lowering the software's quality score; a deduplication rule that deleted all 22 installation pages of a package registered twice in the portal; and a vector search index that quietly stopped returning some relevant passages as it grew.
How well did AI retrieval work on the MateriApps portal?
Mixed. On template questions containing the exact application name, dictionary routing achieved a perfect hit@10 of 1.000. On paraphrased questions without the name, it offered no benefit over plain vector search (hit@10 about 0.62-0.67 in both Japanese and English). Raising the HNSW search-breadth parameter (ef_search) from 100 to 500 recovered recall lost as the index scaled (Japanese hit@5: 0.914 → 0.949).
Is the paper's code and data available?
Data largely yes: the catalog export, the 49,485-passage MateriApps corpus and vector index, and a validation script are archived in the ISSP Data Repository (catalog data under CC BY 4.0; crawled third-party pages included only for verification). Code largely no: the catalog generator, retrieval agent, and evaluation question sets are not publicly released and are available only from the corresponding author on request.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Figures 1-5 are present only as extracted captions and inline labels (e.g., Figure 4's bar counts read as text); the rendered figures were not available, so layout claims rest on captions.
- The metadata-supplied abstract ends mid-sentence, but the embedded full text runs complete through the references and Tables 1-2; no content was treated as truncated.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.