TaxCE : A Framework for Automated Taxonomy Construction and Evaluation at Scale
Sandeep Sricharan Mukku, Albert Aristotle Nanda, Rohit Pyati
cs.CL
Quick verdict
TaxCE automates multi-level taxonomy construction with EEG metrics, beating six baselines by 11.8–20.5 pp; circularity concerns and no code temper the claims.
Summary
TaxCE is a fully automated framework that turns raw feedback corpora — product reviews, consumer complaints, technical support questions — into multi-level hierarchical taxonomies without seed terms, labels, or ontologies. The pipeline has six stages: LLM-based key information extraction of actionable segment-intent pairs; embedding-based semantic grouping with HDBSCAN; LLM concept generation plus corpus-wide standardization into deduplicated atomic concepts; granular leaf-topic and definition generation; bottom-up hierarchy construction with root-to-leaf LLM validation; and iterative refinement driven by three proposed corpus-grounded metrics — Exclusivity, Exhaustivity, Granularity (EEG) — converging within 1-3 iterations. On Flipkart (~194K reviews, 20% machine-translated into four languages), CFPB (~4M complaints), and AskUbuntu (~167K questions), TaxCE (locally hosted small LLM for extraction; Claude-class LLM downstream) scores 85.7-87.1 E, 76.9-80.5 X, and 75.6-78.2 G, beating six baselines (LDA, hLDA, BERTopic, TopClus, LLM zero-shot, LLM+clustering) by average margins of 11.8/20.5/15.7 points. Ablations attribute the largest exclusivity drop (-14.4) to removing concept standardization. Three annotators rated TaxCE highest on quality, actionability, and navigability (4.2-4.4 vs 3.1-3.5 for the strongest baseline; Fleiss' kappa 0.68-0.74). Key limitations: exhaustivity and granularity are measured against a reference set produced by TaxCE's own extraction stage using the same embedding model as the pipeline — a partial circularity the authors defer to future work; no significance testing or run variance is reported; no code is released; and human evaluation covers only three methods.
Review complete
Assessment at a glance
5
risk flags identified
● high 0
● medium 3
● low 2
5 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- TaxCE constructs multi-level hierarchical taxonomies fully automatically from raw text without seed terms, labeled data, or domain ontologies.
- The EEG metrics are the first formal, corpus-grounded definitions of exclusivity, exhaustivity, and granularity, and integrated into a refinement loop that converges within 1-3 iterations.
- TaxCE outperforms six baselines across three datasets with average improvements of 11.8/20.5/15.7 percentage points in E/X/G over the strongest baseline (LLM+Clustering).
- Concept standardization and iterative refinement make complementary, synergistic contributions, with standardization chiefly driving exclusivity (+14.4) and refinement chiefly driving exhaustivity (+7.6).
- Human evaluators (n=3) rate TaxCE taxonomies substantially higher on quality, actionability, and navigability (4.2-4.4 vs 3.1-3.5; Fleiss' kappa 0.68-0.74).
Strengths
Formal corpus-grounded EEG metrics with unusually thorough sensitivity analyses
Equations 1-3 give explicit definitions of Exclusivity, Exhaustivity, Granularity; Appendix I sweeps the coverage threshold tau-x over 0.50-0.80 (rankings stable), Appendix K tests six embedding backbones (cross-embedding Pearson 0.89-0.97 with per-backbone recalibrated tau-x), and Appendix L stress-tests refinement thresholds, disclosing hard ceilings (E approx 87-88, X approx 82-83, G approx 79-80).
Transparent convergence reporting for the metrics-in-the-loop loop
Appendix F, Table 8 reports iteration-wise E/X/G per dataset with the corrective action applied at each step (theta-E=83, theta-X=75, theta-G=74), convergence in 1-3 iterations, and discloses that under 5% of runs fail to converge and are returned flagged.
Component-level ablations plus stage-wise validation
Exemplary appendix documentation
Candid limitations section
Weaknesses
Partial evaluation circularity touching two of three headline metrics
No inferential statistics anywhere
Baseline set omits the taxonomy-construction literature the paper critiques
No code, configs, or outputs released
Minor internal numerical inconsistency
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
TaxCE (arXiv 2608.30614, three Amazon authors) targets fully automated construction of multi-level hierarchical taxonomies from raw feedback corpora: Flipkart reviews (~194K, 20% machine-translated into DE/IT/FR/ES at 5% each), CFPB complaints (~4M), and AskUbuntu (~167K). It claims three contributions: (1) a six-stage pipeline (KIE, semantic grouping, concept generation/standardization, granular topic+definition generation, bottom-up hierarchy with root-to-leaf validation, iterative refinement); (2) formal corpus-grounded EEG metrics (Equations 1-3); (3) the first systematic benchmarking of taxonomy generation under EEG, with average gains of 11.8/20.5/15.7 points over the strongest baseline (LLM+Clustering, Table 1). This is an industrial method paper: a two-tier LLM deployment (locally hosted <=9B models for extraction per Appendix E; Claude-3.5-Sonnet downstream), with cost analysis of 1M documents in under 10 hours (Appendix N).
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 5 strengths & 5 weaknesses, analyzed in detail
- 5 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What is TaxCE and what problem does it solve?
TaxCE is a fully automated framework from Amazon researchers that converts large volumes of unstructured feedback text (product reviews, consumer complaints, support questions) into multi-level hierarchical topic taxonomies, without seed words, labeled data, or a predefined ontology. It addresses three gaps in prior work: shallow hierarchies, neglected long-tail topics, and the absence of rigorous evaluation methods.
What are the EEG metrics in the TaxCE paper?
EEG stands for Exclusivity, Exhaustivity, and Granularity — three corpus-grounded scores on a 0-100 scale. Exclusivity measures semantic distinctiveness between leaf topics via pairwise embedding similarity; Exhaustivity measures the fraction of corpus-extracted actionable segments covered by at least one leaf topic (similarity threshold 0.65); Granularity measures how closely the best-matching leaf topic represents each segment. They are also used inside the system to iteratively fix the taxonomy until thresholds are met.
How much better is TaxCE than existing methods?
Across three public datasets (Flipkart reviews, CFPB complaints, AskUbuntu questions), TaxCE scores roughly 76-87 on the three metrics versus substantially lower scores for six baselines (LDA, hLDA, BERTopic, TopClus, LLM zero-shot, LLM+clustering), with average gains of 11.8, 20.5, and 15.7 percentage points in exclusivity, exhaustivity, and granularity over the strongest baseline. Note the metrics were computed against a reference set produced by TaxCE's own extraction stage, which the reviewers flag as a partial circularity.
Is TaxCE code or data publicly available?
The paper does not mention any released code, configurations, or outputs. It does publish full prompt templates (Appendix O), model choices (small local LLMs for extraction, Claude-3.5-Sonnet-class API for downstream stages), and the three evaluation datasets are public, but exact reimplementation and replication of the reported numbers is not straightforward.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Figures 1-5 were extracted as garbled or unlabeled number sequences; figure-derived values cited in this review were taken from captions, axis-label text, and appendix prose rather than the plots themselves.
- The provided text appears complete through Appendix P.6 (ends with a finished sentence); no truncation was detected, and all cited tables (1-13) were present with readable values.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.