LLM-Enhanced Dual-Branch Learning for Large-Scale Multi-Label Text Classification

Summary

The paper introduces DualMLC, a dual-branch framework for large-scale (extreme) multi-label text classification that processes each document through two heterogeneous encoders: a frozen Qwen2.5-7B decoder-only model adapted with LoRA (rank 16 on attention projections) and a fully fine-tuned BERT-base-uncased encoder. Each branch tokenizes, encodes, and pools the document independently (masked mean vs. CLS) and scores the shared label space with its own linear classifier; the two logit vectors are fused as αzq+(1−α)zb with α=0.6, and both branches train with separate binary cross-entropy losses. On EURLex-4K, Wiki10-31K, and AmazonCat-13K, DualMLC reports the best P@1/3/5 among thirteen baselines — e.g., 90.78/81.11/71.94 on Wiki10-31K versus MatchXML's 89.30/80.45/70.89 — with per-cell gains of 0.35–1.48 points. An ablation on Wiki10-31K shows the Qwen branch alone reaches 90.02 P@1 and BERT 88.74, while fusion adds a further 0.76, which the authors read as evidence of representational complementarity. Sensitivity studies cover fusion weight, pooling, layer averaging, sequence length, and LoRA rank, all with small variation. Main limitations: ablations and apparent hyperparameter selection are confined to the Wiki10-31K test set; results use a single seed without significance testing; no homogeneous-ensemble control isolates heterogeneity from generic ensembling gains; no efficiency comparison is provided despite the 7B backbone; and recent LLM-based XMC baselines are absent from Table I.