CPAE: Contrastive predictive autoencoder for unsupervised pre-training in health status prediction

Fully-supervised learning approaches have shown promising results in some health status prediction tasks using Electronic Health Records (EHRs). These traditional approaches rely on sufficient labeled data to learn from. However, in practice, acquiring large-scaled labeled medical data for various prediction tasks is often not feasible. Thus, it is of great interest to utilize contrastive pre-training to leverage the unlabeled information.

In this work, we propose a novel data-efficient framework, contrastive predictive autoencoder (CPAE), to first learn without labels from the EHR data in the pre-training process, and then fine-tune on the downstream tasks. Our framework comprises of two parts: (i) a contrastive learning process, inherited from contrastive predictive coding (CPC), which aims to extract global slow-varying features, and (ii) a reconstruction process, which forces the encoder to capture local features. We also introduce the attention mechanism in one variant of our framework to balance the above two processes.

Experiments on real-world EHR dataset verify the effectiveness of our proposed framework on two downstream tasks (i.e., in-hospital mortality prediction and length-of-stay prediction), compared to their supervised counterparts, the CPC model, and other baseline models.

By comprising of both contrastive learning components and reconstruction components, CPAE aims to extract both global slow-varying information and local transient information. The best results on two downstream tasks are all achieved by CPAE. The variant AtCPAE is particularly superior when fine-tuned on very small training data. Further work may incorporate techniques of multi-task learning to optimize the pre-training process of CPAEs. Moreover, this work is based on the benchmark MIMIC-III dataset which only includes 17 variables. Future work may extend to a larger number of variables.