A novel transformer-based semantic feature extraction method for multi-label text classification

Multi-label text classification is a critical task in natural language processing, in which each document may belong to multiple categories. This setting is challenging, as it involves complex label dependencies and requires extracting fine-grained semantic features for each label. In this paper, we propose the TMSFE, a new Transformer-based semantic feature extraction method for multi-label text classification, which integrates label-specific query embeddings with a multi-head attention mechanism to extract discriminative features for each potential label and leverages a Latent semantic space to enhance the efficiency of feature extraction. Unlike conventional single-label classifiers or flat multi-label methods, the proposed model designs a DeBERTaV3-based Transformer encoder to jointly model the document and label semantics. Additionally, the proposed SimCSE-Based latent semantic space module projects text and label representations into a shared latent semantic space to enhance feature extraction efficiency. And a sigmoid-based multi-label classification head is applied to the extracted features. Results show that the proposed TMSFE consistently outperforms baseline models, achieving lower Hamming loss and higher feature extraction accuracy.