Pretrained transformer framework on pediatric claims data for population specific tasks.

Area of Science:

• * Medical informatics
• * Machine learning in healthcare
• * Pediatric research

Background:

• * Electronic health records (EHR) enable data-driven healthcare research.
• * Deep learning models require large datasets, which are often scarce for specific patient populations.
• * Training models on limited cohorts challenges the development of population-specific predictive tools.

Purpose of the Study:

• * To present the Claim Pre-Training (Claim-PT) framework for effective deep learning model training on scarce population-specific data.
• * To demonstrate knowledge transfer from a large general dataset to specialized tasks with minimal data.
• * To address the challenge of data scarcity in training deep learning models for pediatric healthcare.

Main Methods:

• * Developed a generic pre-training model (Claim-PT) trained on an extensive pediatric claims dataset.
• * Employed a discriminative fine-tuning approach on population-specific tasks.
• * Utilized task-aware fine-tuning with minimal parameter modification and unchanged model architecture.

Main Results:

• * Claim-PT framework outperformed tailored task-specific models by over 10% in performance on two downstream tasks.
• * Demonstrated effective knowledge transfer from a large general dataset to smaller, specific cohorts.
• * Showcased potential for cross-institutional knowledge transfer in healthcare AI models.

Conclusions:

• * The Claim-PT framework effectively mitigates data scarcity issues in training deep learning models for specialized medical tasks.
• * Pre-training on large datasets followed by fine-tuning enables robust model development even with small patient cohorts.
• * This approach holds promise for advancing AI in healthcare through efficient knowledge transfer and improved model performance.