Blood Glucose Prediction Algorithms Require Clinically Relevant Performance Criteria Beyond Accuracy.

Area of Science:

• Biomedical Engineering
• Data Science in Healthcare
• Diabetes Technology

Background:

• Root Mean Squared Error (RMSE) is standard for evaluating blood glucose prediction algorithms.
• RMSE prioritizes accuracy within the target range, potentially overlooking critical glycemic events like hypoglycemia or hyperglycemia.
• This bias can lead to the selection of suboptimal models for proactive diabetes management.

Purpose of the Study:

• To investigate the limitations of RMSE in evaluating blood glucose prediction models.
• To introduce a novel metric, the Composite Glucose Prediction Metric (CGPM), for more clinically relevant evaluations.
• To develop and test a custom loss function for optimizing models towards critical event prediction.

Main Methods:

• Developed the Composite Glucose Prediction Metric (CGPM) integrating RMSE, temporal gain, and geometric mean.
• Designed a custom loss function to prioritize clinically critical glycemic event predictions.
• Applied Pareto frontier analysis to compare different prediction models on the OhioT1DM dataset.

Main Results:

• Models optimized solely for RMSE performed poorly in predicting critical glycemic events.
• A ridge regression model trained with the custom loss function demonstrated improved prediction of critical events.
• The study confirmed RMSE's bias towards target-range predictions and highlighted the effectiveness of clinically weighted optimization.

Conclusions:

• Existing accuracy metrics like RMSE are insufficient for optimal diabetes management decision support.
• The proposed CGPM offers a more comprehensive framework for evaluating blood glucose prediction algorithms.
• Clinically informed optimization strategies are crucial for developing reliable diabetes management tools.