Cleaning method for abnormal energy big data based on sparse self-coding.

Area of Science:

• Data Science
• Energy Systems Analysis
• Machine Learning

Background:

• Abnormal energy big data poses challenges for accurate analysis and anomaly detection.
• Existing methods struggle with interference from abnormal data, impacting reliability.
• Dynamic carbon emissions influence energy data, necessitating advanced cleaning techniques.

Purpose of the Study:

• To propose and validate a sparse self-coding method for cleaning abnormal energy big data.
• To enhance the accuracy of anomaly detection and data cleaning processes.
• To address the interference of abnormal data in energy big databases.

Main Methods:

• Multi-criteria evaluation for abnormal data detection and spectral feature analysis.
• Chaotic time series reconstruction, robust local weighted regression, and sparse self-coding for feature decomposition.
• Adaptive segmentation based on periodicity, AFCM algorithm for anomaly index, and LOF-based evaluation model.

Main Results:

• Achieved an error detection rate of 0.24% and a missing detection rate of 0.27%.
• Demonstrated a cleaning rate of 99.49% for abnormal energy big data.
• Completed data cleaning in less than 2 seconds, showcasing high efficiency.

Conclusions:

• The proposed sparse self-coding method is highly effective for cleaning abnormal energy big data.
• The method significantly improves anomaly detection accuracy and reduces data interference.
• The approach offers a fast and efficient solution for real-world energy data cleaning applications.