Improving Transformer Performance for French Clinical Notes Classification Using Mixture of Experts on a Limited...

Area of Science:

• Natural Language Processing (NLP)
• Machine Learning
• Clinical Informatics

Background:

• Transformer models excel in NLP but struggle with small-scale clinical text classification due to computational demands.
• Existing biomedical pre-trained models (e.g., CamemBERT-bio, DrBERT) are computationally intensive, limiting their use in hospital settings.
• Need for efficient NLP solutions for French clinical narratives within resource-constrained hospital environments.

Purpose of the Study:

• To develop and evaluate a customized Mixture of Expert (MoE) Transformer model for classifying small-scale French clinical texts.
• To address challenges of limited data and low-resource computation for in-house hospital applications.
• To provide a practical NLP alternative for clinical text analysis in resource-limited settings.

Main Methods:

• Development of a customized MoE-Transformer architecture.
• Training and evaluation on small-scale French clinical texts from CHU Sainte-Justine Hospital.
• Comparative analysis against DistillBERT, CamemBERT, FlauBERT, and standard Transformer models.

Main Results:

• The MoE-Transformer achieved 87% accuracy, 87% precision, 85% recall, and 86% F1-score.
• Outperformed DistillBERT, CamemBERT, FlauBERT, and Transformer models on the same dataset.
• Achieved training speeds at least 190 times faster than comparable biomedical BERT models, despite slightly lower performance.

Conclusions:

• The customized MoE-Transformer offers a computationally efficient and effective solution for classifying small French clinical texts.
• Presents a viable alternative to high-resource biomedical BERT models in constrained clinical settings.
• Demonstrates significant potential for integration into clinical decision support systems, particularly in pediatric intensive care units.