Enhancing meteorological data reliability: An explainable deep learning method for anomaly detection.

Area of Science:

• Meteorological Science
• Data Science
• Artificial Intelligence

Background:

• Accurate meteorological data is vital for human activities, but instrumental errors and unstable sensor resources can lead to significant data deviations.
• Detecting subtle, systematic anomalies in meteorological data is challenging, requiring advanced analytical methods beyond simple outlier identification.

Purpose of the Study:

• To develop an interpretable deep learning method for prompt and accurate anomaly detection in meteorological observational data.
• To improve the reliability of meteorological data for applications in agriculture, climate observation, and disaster prevention.

Main Methods:

• An autoencoder (AE) was used for initial anomaly detection by identifying data with high reconstruction errors.
• SHapley Additive exPlanations (SHAP) were employed to evaluate the importance of individual meteorological elements in flagged anomalous data.
• A K-sigma based threshold method was utilized for automatic anomaly threshold delineation, adapting to site-specific data characteristics.
• Bayesian optimization (BO) was applied to fine-tune hyperparameters, optimizing the deep learning model's structure and anomaly detection accuracy.

Main Results:

• The proposed method effectively identifies anomalies in multidimensional meteorological datasets.
• SHAP analysis provides interpretability by highlighting key contributing meteorological elements to detected anomalies.
• The integrated approach, combining AE, SHAP, and BO, enhances the accuracy and robustness of anomaly detection.

Conclusions:

• The developed interpretable deep learning model offers a powerful tool for early and accurate detection of anomalies in meteorological data.
• This method addresses the challenge of identifying subtle, emerging anomalies, improving data quality for critical applications.
• The findings have significant practical implications for enhancing agricultural production, climate monitoring, and disaster preparedness.