A probabilistic framework for identifying anomalies in urban air quality data.

Area of Science:

• Environmental Science
• Data Science
• Atmospheric Chemistry

Background:

• Air quality data requires systematic processing to be useful for decision-making.
• Ground-based monitoring data often contains outliers, which can be error-based or event-based.
• Error-based outliers (e.g., instrument malfunction, sensor drift) are noise and must be removed, unlike meaningful event-based outliers.

Purpose of the Study:

• To develop and validate a robust methodology for detecting error-based outliers in air quality data.
• To specifically target particulate matter (PM2.5 and PM10) data from monitoring sites in Delhi.
• To improve the reliability of air quality data for subsequent analysis and modeling.

Main Methods:

• Developed a non-linear filtering approach to model air quality data.
• Calculated residuals between observed and predicted values.
• Utilized Z-scores to assess residual probabilities and flag outliers based on a sensitivity-tested threshold.

Main Results:

• Identified four distinct types of error-based outliers in PM2.5 and PM10 data: extreme values, constant readings/low variance, periodic self-calibration issues, and PM2.5/PM10 ratio anomalies.
• Successfully flagged outliers using the developed Z-score based probability method.
• Demonstrated the method's effectiveness on data with less than 5% missing values.

Conclusions:

• The proposed methodology effectively detects error-based outliers in air quality data.
• Refined data quality is essential for accurate air quality modeling and reliable statistical/machine learning applications.
• This approach enhances the integrity of environmental data for informed decision-making.