Generalizing machine learning models from clinical free text.

Area of Science:

• Artificial Intelligence in Healthcare
• Medical Informatics
• Machine Learning Generalizability

Background:

• Healthcare artificial intelligence (AI) models often struggle with generalizability across different institutions.
• Developing AI that performs reliably in diverse clinical settings is crucial for widespread adoption.
• Strategies to improve the robustness of AI models trained on medical data require thorough investigation.

Purpose of the Study:

• To evaluate methods for enhancing the generalizability of AI models in healthcare.
• To assess the impact of text preprocessing techniques on model performance.
• To compare single-institution versus multiple-institution data models and explore data divergence metrics.

Main Methods:

• Deep neural network models were developed to classify anesthesiology Current Procedural Terminology codes from medical free text.
• Three levels of text preprocessing were analyzed: minimal, automated (cSpell), and physician-reviewed.
• Kullback-Leibler Divergence and k-medoid clustering were employed to assess model performance and data divergence.

Main Results:

• Single-institution models achieved high internal accuracy but poor external generalizability.
• Text preprocessing demonstrated minimal impact on model performance.
• Multi-institution models improved external generalizability but had lower internal accuracy compared to single-institution models.
• Kullback-Leibler Divergence showed a strong correlation with model performance (R²=0.41), outperforming other metrics.

Conclusions:

• Medical free text preprocessing has limited utility in improving AI model generalization.
• While single-institution models excel internally, multi-institution models offer better generalizability.
• Kullback-Leibler Divergence serves as a valuable heuristic for evaluating AI model generalizability in healthcare.