Data pipeline for real-time energy consumption data management and prediction

Area of Science:

• Data Engineering
• Machine Learning Operations (MLOps)
• Energy Systems

Background:

• Efficient data pipelines are critical for real-time data utilization across industries.
• Energy consumption management systems require robust data processing and prediction capabilities.

Purpose of the Study:

• To propose a machine learning operations-centric data pipeline for energy consumption management.
• To integrate real-time data management and prediction with machine learning models.
• To evaluate the pipeline's efficiency and scalability.

Main Methods:

• Developed a data pipeline architecture using Kafka, InfluxDB, Telegraf, Zookeeper, and Grafana.
• Implemented and compared two time-series prediction models: Long Short-Term Memory (LSTM) and Seasonal Autoregressive Integrated Moving Average (SARIMA).
• Optimized Telegraf configuration to measure end-to-end processing time.

Main Results:

• The pipeline achieved an average end-to-end processing time of 0.39s for 10,000 records and 1.26s for 100,000 records.
• Demonstrated processing speeds 30.69-90.88 times faster than existing Python-based approaches.
• Showcased a 3.07 times reduction in increased overhead when data volume increased tenfold.
• Identified a speed-accuracy trade-off: SARIMA is faster, while LSTM offers higher prediction accuracy.

Conclusions:

• The proposed pipeline is efficient and scalable for real-time energy consumption management.
• The architecture effectively integrates data management, prediction, and MLOps principles.
• The system provides a significant performance improvement over traditional methods.