SPIRIT: Structural Entropy Guided Prefix Tuning for Hierarchical Text Classification

Area of Science:

• Natural Language Processing
• Machine Learning
• Artificial Intelligence

Background:

• Hierarchical text classification (HTC) presents challenges due to complex label dependencies.
• Prompt tuning methods for HTC show limitations in maintaining model attention across language model layers.
• Existing prompt tuning struggles with effectively leveraging the intricate structure of label hierarchies.

Purpose of the Study:

• To address the limitations of current prompt tuning in hierarchical text classification.
• To introduce a novel method that enhances language model attention to hierarchical structures.
• To improve the performance of hierarchical text classification by better integrating label hierarchy information.

Main Methods:

• Structural Entropy Guided Prefix Tuning (SPIRIT) is proposed, utilizing structural entropy minimization.
• Essential label hierarchy structures are extracted and decoded as prefixes for language models.
• A depth-wise reparameterization strategy is employed to optimize and propagate prefixes through all LM layers.

Main Results:

• SPIRIT demonstrates state-of-the-art performance on four widely-used hierarchical text classification datasets.
• The method effectively prompts all intermediate layers of the language model, overcoming previous attention decay issues.
• Enhanced optimization and prefix propagation lead to significant performance gains.

Conclusions:

• SPIRIT offers a superior approach to hierarchical text classification by effectively integrating label hierarchy information.
• The proposed method overcomes limitations of previous prompt tuning techniques in HTC.
• SPIRIT sets a new benchmark for performance in hierarchical text classification tasks.