Bad and Good Errors: Value-Weighted Skill Scores in Deep Ensemble Learning.

Area of Science:

• Data Science
• Machine Learning
• Predictive Analytics

Background:

• Forecast verification traditionally relies on quality-based skill scores.
• Assessing predictive model performance is crucial for prognostic models.

Purpose of the Study:

• To propose a novel forecast verification approach focusing on prediction value, not just quality.
• To introduce value-weighted skill scores that account for the severity of forecast errors.
• To develop an ensemble strategy to optimize both quality-based and value-weighted skill scores.

Main Methods:

• Developed a strategy to assess forecast error severity based on event context (isolated vs. consecutive).
• Introduced value-weighted skill scores, prioritizing prediction value.
• Implemented an ensemble strategy to independently maximize quality-based and value-weighted skill scores.
• Tested the approach on deep learning binary classification predictions for pollution, space weather, stock prices, and IoT data.

Main Results:

• The ensemble strategy for maximizing value-weighted skill scores generally improved forecast value.
• Both forecast value and quality were enhanced by the proposed approach.
• Demonstrated effectiveness across diverse applications including environmental and financial forecasting.

Conclusions:

• Value-weighted skill scores offer a more nuanced approach to forecast verification.
• The proposed ensemble strategy effectively improves predictive model performance.
• This method enhances the practical utility of forecasts by considering their real-world impact.