Exploring the Quality of Dynamic Open Government Data Using Statistical and Machine Learning Methods.

Area of Science:

• Data Science
• Open Government Data
• Traffic Engineering

Background:

• Dynamic Open Government Data (OGD), including environmental and traffic data, is crucial for data intelligence applications.
• These dynamic datasets are susceptible to data quality errors from sensor failures and network issues.
• Assessing the quality of dynamic OGD is essential for reliable data-driven insights and applications.

Purpose of the Study:

• To explore and assess the data quality of Dynamic Open Government Data.
• To investigate missing values and anomalies in traffic data from the Greek OGD portal.
• To identify the types and extent of data quality issues in dynamic OGD.

Main Methods:

• Utilized statistical and machine learning methods for data quality assessment.
• Employed traffic flow-speed correlation, seasonal-trend decomposition, and Isolation Forest (iForest) for anomaly detection.
• Applied explainable artificial intelligence (XAI) to understand iForest anomaly classification.

Main Results:

• 20.16% of traffic observations were missing.
• 50% of sensors exhibited 15.5% to 33.43% missing values.
• Average anomaly rate per sensor was 71.1%, with significant detection by both seasonal-trend decomposition (12.6%) and iForest (11.6%).

Conclusions:

• Dynamic OGD, specifically traffic data, suffers from substantial missing values and anomalies.
• The findings indicate critical data quality challenges for developing reliable data intelligence applications.
• This study represents a novel exploration into the quality of dynamic Open Government Data.