Integrating Confidence, Difficulty, and Language Model Calibration for Better Explainability in Clinical Documents...

Area of Science:

• Artificial Intelligence
• Machine Learning
• Clinical Informatics

Background:

• Increasing interest in deep learning for clinical document annotation.
• Challenges in interpretability and transparency of complex models.

Purpose of the Study:

• Improve interpretability of transformer models for clinical document annotation.
• Evaluate explainability of deep learning models using coded clinical data from death certificates.
• Leverage calibrated confidence, saliency maps, and instance difficulty for model interpretation.

Main Methods:

• Utilized disease language bidirectional encoder representations from transformers (DiBERT) pretrained on International Statistical Classification of Diseases and Related Health Problems (ICD) data.
• Analyzed death certificate data (2014-2017) with textualized ICD codes.
• Applied temperature scaling for model calibration and Variance of Gradients for instance difficulty analysis.
• Employed saliency maps (Integrated Gradients) for word-level attribution.

Main Results:

• Achieved high accuracy (0.990) in predicting the cause of death.
• Demonstrated good initial model calibration (Expected Calibration Error: 1.40).
• Showcased effectiveness of Variance of Gradients in analyzing model behavior and identifying out-of-distribution cases.

Conclusions:

• Enhanced interpretability and explainability improve the practical utility of deep learning in clinical document annotation.
• Proposed approaches support trustworthy application of AI in critical medical settings.
• Highlighted the need to address data limitations for robust performance, especially in complex cases.