Attention Mechanisms in Clinical Text Classification: A Comparative Evaluation.

Area of Science:

• Artificial Intelligence
• Natural Language Processing
• Biomedical Informatics

Background:

• Attention mechanisms are crucial for neural networks in biomedical text classification.
• Label-wise attention mechanisms enhance model discrimination using label-specific information.
• Current methods arbitrarily select reference information, impacting model performance.

Purpose of the Study:

• To evaluate label-wise attention initialized with implicit or explicit reference information.
• To compare label-wise attention against baseline methods across various text-encoder architectures.
• To introduce an extension of label-wise attention incorporating code hierarchy information.

Main Methods:

• Experiments were conducted on the MIMIC III dataset.
• Evaluated label-wise attention with implicit/explicit reference information against target-attention and text-encoder specific methods.
• Tested across four text-encoder architectures: CNN, two RNNs, and Transformer.
• Introduced a hierarchical label-wise attention extension.

Main Results:

• Pretrained reference information and hierarchical design significantly improved classification performance.
• Performance gains were less pronounced on larger datasets and label spaces.
• Attention energy scores were used to analyze performance and interpretability differences.

Conclusions:

• Initializing label-wise attention with pretrained reference information and hierarchical structure enhances biomedical text classification.
• The effectiveness of these enhancements varies with dataset size and label space complexity.
• Attention mechanism analysis provides insights into model behavior across different architectures.