A framework for exploration and cleaning of environmental data--Tehran air quality data experience

Area of Science:

• Environmental Science
• Data Science
• Air Quality Monitoring

Background:

• Managing large environmental datasets presents significant challenges.
• Data cleaning and exploration are crucial for accurate environmental analysis.
• Air quality data, specifically particulate matter (PM10), requires robust management techniques.

Purpose of the Study:

• To present a novel framework for exploring and cleaning large environmental datasets.
• To apply this framework to Tehran's air quality data (PM10) from 1996-2013.
• To develop and utilize a spatiotemporal screening tool (SST) for outlier detection.

Main Methods:

• Data acquisition and database development for PM10 concentrations.
• Implementation of data cleaning algorithms, including plausibility range checks.
• Development and application of the spatiotemporal screening tool (SST) for outlier identification, considering dust storm effects.

Main Results:

• The mean PM10 concentration in Tehran from 1996-2013 was 88.96 µg/m³ (raw) and 88.41 µg/m³ (cleaned).
• Data cleaning reduced the standard deviation of PM10 concentrations from 90.86 µg/m³ to 61.64 µg/m³.
• The SST identified 69% of the 5.7% unacceptable outliers, with 29% of outliers falling outside the plausibility range.

Conclusions:

• A novel framework for cleaning large environmental spatiotemporal datasets has been developed.
• The study provides a comprehensive analysis of PM10 levels in Tehran from 1996-2013.
• The framework is recommended for application on large spatiotemporal databases, particularly in developing nations.