Autoresearch for Marketplace Catalogs: From Legacy Forms to AI-Native Matching
Kartik Ravisankar, Hojat Abdolanezhad, Daniel Capo, Sang Su Lee, Shishir Dash, Vijay Anand Raghavan
cs.AI
Quick verdict
Deploys an LLM autoresearch loop rebuilding marketplace taxonomies across 132 occupations; candid about judge-human disagreement (kappa≈0), but quality evidence stays self-referential.
Summary
This industrial experience paper from Thumbtack presents a per-occupation "autoresearch" loop that reconstructs provider-side preference catalogs to replace legacy request-form Q&A in a two-sided services marketplace. Each of 132 occupations is generated independently: a GPT-5-4 generator proposes tag sets, a GPT-5-4-mini judge scores them on six rubrics (composite /15), a seven-persona Claude Sonnet 4.6 critic panel adds weighted penalties, and an editor makes one prompt edit per iteration (budget B=5), followed by an LLM parity-mapping stage linking tags back to legacy Q&A. Reported fidelity: 73.3% of legacy answers map directly, 20.8% are classed acceptable non-tags, 6.0% are misses (pooled coverage 92.5%). From a 14-day cohort (1,840 pros, 9.3M filter evaluations), the loop's median judge score rose from 11.36 to 13.84; a production audit found three deprecated tags driving ~486K filter events and a 40.66% missing-canonical-tag failure attributed to deployment seeding. Limitations: quality gains are measured by the system's own judge, which agrees with human PM verdicts at chance level (kappa≈0); no outcome-level comparison to legacy forms; all data proprietary; several validations (full-cohort critic ablation, dual-evaluator) remain planned rather than done.
Review complete
Assessment at a glance
5
risk flags identified
● high 1
● medium 3
● low 1
5 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- Per-occupation independent generation avoids the semantic interference of a single global hierarchy and yields unambiguous attribution of deployment mismatches (§1, §6.3).
- A recalibrated six-rubric LLM judge plus a seven-persona critic panel with weighted penalties (no hard vetoes) produces deployable catalogs, lifting median E composite from 11.36 to 13.84 across 132 occupations (§6.5).
- Parity mapping by inferring the intended provider attribute (Step-Back style) achieves 92.5% pooled coverage of mappable legacy answers, with only 6.0% regrettable misses, making legacy-form retirement defensible (§1, §4.6).
- The judge's discrete verdict does not predict human reviewer actions (kappa ≈ 0), so it is used as a ranking signal with mandatory human sign-off (§6.5, Table 4).
- Production monitoring surfaced a catalog-hygiene gap (3 deprecated tags, ~486K filter events) and a 40.66% missing-canonical-tag failure rate attributable to deployment-side seeding, not catalog quality (§6.3).
Strengths
Real production deployment with verifiable operational telemetry
132 occupations live since April 2026; §6.3 reports a concrete 14-day cohort (1,840 pros, 9,324,798 filter evaluations, Table 2) and a catalog-hygiene find (3 deprecated tags → ~486K filter events), which is the kind of evidence only an operating system can produce.
Deliberately reported negative result on judge-vs-human agreement
§6.5/Table 4 shows raw E-verdict vs PM-consensus agreement of 77.2% over 4,112 tags is entirely base-rate (model auto-approves 97.3%, reviewers keep 79.1%), Cohen's kappa ≈ 0. The authors use this to justify E as a ranking signal only, with mandatory human sign-off — an unusually honest evaluation posture.
Clean attribution discipline between catalog quality and deployment outcome
Marginal-contribution analysis of the critic panel
Candid failure post-mortems
Weaknesses
Core quality evidence is self-referential
No outcome-level comparison against legacy forms
Fidelity headline depends on an LLM-judged exclusion bucket
Ablations are thin and partly deferred
Proprietary stack, unreproducible, with internal inconsistencies
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
This KDD 2026 workshop paper (industry track in spirit, accepted at the Workshop on Evaluation and Trustworthiness of Agentic AI) reports a **deployed system**, not an algorithmic contribution. Thumbtack's marketplace is migrating from deterministic request-form Q&A intake to AI-native probabilistic matching; the paper describes the "autoresearch" loop that regenerates the provider-side preference taxonomy this migration requires — one occupation at a time, 132 occupations live since April 2026.
The four claimed contributions (§1): (i) the per-occupation preference-tag catalog as the unit of optimization, contrasted with occupation-hierarchy construction (CLIMB [9]); (ii) independent per-occupation generation loops; (iii) marketplace-grounded evaluation via a six-rubric LLM judge plus a seven-persona critic panel with weighted penalties and no hard vetoes; (iv) legacy-Q&A parity mapping as a first-class pipeline stage. The headline fidelity claim: 73.3% of legacy answers map directly, 20.8% are "acceptable non-tags", 6.0% regrettable misses — 92.5% pooled coverage of what should map.
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 5 strengths & 5 weaknesses, analyzed in detail
- 5 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What does the paper 'Autoresearch for Marketplace Catalogs' actually do?
It describes a deployed system at Thumbtack that uses large language models to automatically rebuild the provider-preference catalog of a services marketplace. Instead of one global taxonomy, each of 132 service occupations (plumbing, cleaning, photography, etc.) is independently reconstructed by an iterative loop: an LLM generates candidate tag sets, another LLM scores them on six quality rubrics, a panel of seven LLM 'critic' personas applies penalties, and an editor LLM refines the generation prompt over up to 5 iterations. A separate mapping step then links the new tags back to the old question-and-answer forms so humans can verify nothing was lost before the old forms are retired.
How well did the system perform in production?
The authors report 73.3% of legacy form answers map directly to a generated tag, 20.8% are judged intake-only details that need no tag, and 6.0% are real misses, giving 92.5% coverage of what should map. The median LLM-judge score of kept catalogs rose from 11.36 to 13.84 out of 15 across 132 occupations. However, the paper also shows its judge agrees with human product-manager verdicts only at chance level (kappa ≈ 0), so every catalog still required human sign-off before deployment.
Did replacing Q&A forms with LLM-generated tags improve marketplace outcomes?
Not yet demonstrably. In a 14-day post-launch cohort (1,840 pros, 9.3 million filter evaluations), 40.66% of evaluations failed because providers were never seeded with the required canonical tag — a deployment wiring gap, not a catalog gap. The authors explicitly state that an outcome-level comparison against the legacy forms remains future work, and their longer-window signals (8–10 weeks) are described as directional, not causal.
Can the system be reproduced from the paper?
Not fully. No code, prompts, or penalty formulas are released (details are in unspecified supplementary material), and all data comes from Thumbtack's proprietary production logs. The architecture (per-occupation loops, cross-family model stacking, parity mapping) is described conceptually, but the exact rubric texts, critic weights, and judge calibration procedure are not in the paper's body.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- The provided text is complete through the References; no content appears cut off. Supplementary material (penalty formulas, Accounting walkthrough, critic caps) is referenced by the paper but not included in the extraction, so those details could not be verified.
- Model names 'GPT-5-4' / 'GPT-5-4-mini' are reported verbatim from the text; they could not be cross-checked against any external catalog under the no-network constraint.
- The 'MEMORY_ONLY typing primitive' in §8 appears nowhere else in the extracted text; flagged as an internal inconsistency rather than an extraction loss, since the full body and conclusion were present.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.