Adaptive noise-augmented attention for enhancing Transformer fine-tuning on longitudinal medical data

Area of Science:

• Artificial Intelligence
• Machine Learning
• Computational Medicine

Background:

• Transformer models excel in various domains but struggle with clinical predictions from electronic health records (EHR) due to limited data and complex event sequences.
• Self-attention mechanisms in transformers may underperform in capturing subtle dependencies within sparse clinical events under limited supervision.

Purpose of the Study:

• To introduce Adaptive Noise-Augmented Attention (ANAA), a novel fine-tuning technique for transformer models applied to clinical prediction tasks.
• To enhance the modeling of subtle temporal dependencies in longitudinal medical data.

Main Methods:

• ANAA injects adaptive noise into self-attention weights and applies a 2D Gaussian kernel to smooth attention maps.
• This method broadens and refines attention distributions, emphasizing informative clinical events without altering the model architecture or pre-training.
• The technique operates solely during the fine-tuning phase.

Main Results:

• Consistent performance improvements were observed across multiple clinical prediction tasks.
• ANAA effectively shapes learned attention behavior, providing interpretable insights into temporal dependency modeling.
• The approach demonstrates superior performance compared to standard fine-tuning methods.

Conclusions:

• ANAA is a simple yet effective fine-tuning technique for improving transformer-based clinical prediction models.
• The method addresses limitations in handling sparse, event-driven medical data under limited supervision.
• ANAA offers a practical solution for enhancing the interpretability and performance of models using electronic health records.