Unsupervised pre-training of graph transformers on patient population graphs

Area of Science:

• Machine Learning
• Clinical Data Analysis
• Medical Informatics

Background:

• Pre-training methods have advanced various machine learning fields but remain underexplored in clinical data analysis.
• Scarcity of labeled data in clinical settings, especially for rare diseases or small hospitals, limits model performance.
• Unlabeled clinical data offers a valuable resource for improving predictive models.

Purpose of the Study:

• To propose novel unsupervised pre-training techniques for heterogeneous, multi-modal clinical data.
• To enhance patient outcome prediction by adapting masked language modeling (MLM) principles using graph deep learning.
• To introduce a graph-transformer-based network capable of handling diverse clinical data types.

Main Methods:

• Leveraging graph deep learning over population graphs to model patient data.
• Applying masked language modeling (MLM)-inspired pre-training on unlabeled clinical data.
• Developing and utilizing a graph-transformer network for heterogeneous clinical data analysis.
• Evaluating the pre-training methods in self-supervised and transfer learning settings on multiple datasets (TADPOLE, MIMIC-III, Sepsis Prediction).

Main Results:

• The proposed pre-training methods effectively model clinical data at both patient and population levels.
• Significant performance improvements were observed across various fine-tuning tasks on all tested medical datasets.
• The combination of masking-based pre-training and transformer networks successfully extends these techniques to heterogeneous clinical data.

Conclusions:

• Unsupervised pre-training on unlabeled clinical data is beneficial for patient outcome prediction.
• The proposed graph-transformer approach effectively handles heterogeneous clinical data.
• These methods offer a promising direction for improving machine learning applications in clinical data analysis, particularly when labeled data is scarce.