Design of a Spark Big Data Framework for PM_2.5 Air Pollution Forecasting

Area of Science:

• Environmental Science
• Computer Science
• Data Science

Background:

• Air pollution, particularly PM2.5, poses significant health and environmental risks.
• Existing air quality monitoring stations have limitations in deployment, update frequency, and data handling capacity.
• The large volume of real-time air quality data requires advanced computational frameworks for analysis and prediction.

Purpose of the Study:

• To develop an instant prediction architecture for PM2.5 air quality using the Spark big data framework.
• To address the challenges of high computational requirements and large datasets in air quality monitoring.
• To provide short-term PM2.5 forecasts to support timely public health interventions.

Main Methods:

• Implementation of a three-module Spark big data framework for real-time PM2.5 data collection.
• Application of ensemble learning combining Linear Regression, Random Forest, and Gradient Boosting Decision Tree algorithms.
• Prediction of PM2.5 concentration values for the next 30 to 180 minutes with accompanying visualization.

Main Results:

• The proposed Spark big data ensemble prediction model achieved a high performance, with R² up to 0.96 for 30-minute predictions.
• The ensemble model demonstrated superior performance compared to individual machine learning algorithms.
• The system effectively handles large datasets from the LASS community for instant PM2.5 forecasting.

Conclusions:

• The Spark big data framework provides an efficient solution for real-time PM2.5 prediction.
• Ensemble learning significantly enhances the accuracy of short-term air quality forecasting.
• The developed system can assist decision-makers in taking immediate actions to mitigate the impact of poor air quality.