Benchmarking AutoML frameworks for disease prediction using medical claims

Area of Science:

• Health Informatics
• Machine Learning in Healthcare
• Data Science

Background:

• Healthcare datasets are often large and highly imbalanced, posing challenges for predictive modeling.
• Automated Machine Learning (AutoML) offers potential for streamlining model development in clinical settings.

Purpose of the Study:

• To evaluate and compare the performance of three AutoML tools on large, imbalanced healthcare administrative claims data.
• To assess the effectiveness of AutoML in predicting disease outcomes using historical claims data.

Main Methods:

• A large dataset of de-identified administrative claims was generated, including demographics and disease codes.
• Three AutoML tools were trained to predict six disease outcomes and evaluated using various performance metrics.

Main Results:

• AutoML tools improved upon a baseline random forest model but did not significantly differ from each other.
• Models exhibited low area under the precision-recall curve, struggling to balance true positive prediction with high true negative rates.
• Model performance was not correlated with disease prevalence; threshold selection is critical for practical application.

Conclusions:

• Healthcare data's size and imbalance present significant challenges for current AutoML tools.
• Enhancements in scalability, imbalance-handling techniques, and feature engineering are necessary for improved AutoML performance in healthcare.
• No single AutoML tool demonstrated consistent superiority, indicating a need for further development in handling medical claims data.