Few-Shot Learning with Semi-Supervised Transformers for Electronic Health Records

Area of Science:

• Biomedical informatics
• Machine learning in healthcare

Background:

• Electronic Health Records (EHRs) offer vast potential for deep learning in medical prediction.
• Transformer architectures excel with EHRs but struggle with small, specialized datasets.
• Limited data is common in rare diseases, invasive procedures, and specific cohort studies.

Purpose of the Study:

• To develop a semi-supervised transformer-based architecture for improved patient representation learning.
• To address the challenge of rapid performance degradation in deep learning models with small target datasets.
• To enable effective medical prediction in few-shot learning scenarios with limited annotated data.

Main Methods:

• Proposed CEHR-GAN-BERT, a semi-supervised transformer architecture.
• Leveraged both in- and out-of-cohort patients for enhanced representation learning.
• Evaluated on four prediction tasks across three public datasets.

Main Results:

• Achieved performance improvements exceeding 5% across all metrics (AUROC, F1 Score).
• Demonstrated significant gains on tasks with fewer than 200 annotated patients.
• Showcased the model's effectiveness in few-shot learning scenarios.

Conclusions:

• CEHR-GAN-BERT offers a viable solution for medical prediction with limited EHR data.
• The method enhances patient representation learning, crucial for downstream tasks.
• Opens new avenues for leveraging deep learning in data-scarce biomedical applications.