A visual analytic method to investigate patterns and structures of missing data

Area of Science:

• Data Science
• Visual Analytics
• Machine Learning

Background:

• Understanding missing data patterns and structures is crucial for preventing analysis bias and building accurate models.
• Investigating these missing data characteristics presents significant challenges in large datasets.

Purpose of the Study:

• To develop a novel visual analytic method for investigating missing data patterns and structures.
• To address the challenges associated with identifying complex missingness in datasets.
• To improve data quality assessment and facilitate more reliable data analysis.

Main Methods:

• A novel visual analytic method combining purity metrics for block, monotone, and disjoint patterns.
• Integration of seven distinct visualizations within an iterative workflow.
• Application of perceptual discontinuity for enhanced visibility of small values in visualizations.

Main Results:

• Identified a rare data quality issue (0.08% of records) in a UCI dataset, missed by correlation-based methods.
• Corrected reported completeness of two variables in the UCI dataset, revealing 0.02% and 0.3% missing values.
• Reduced 595,171 unique missing value combinations in a hospital dataset to 57 comprehensible patterns.

Conclusions:

• The developed method effectively reveals data quality issues, data dictionary errors, and variable relationships.
• The visual analytic approach aids reproducibility through an explanation graph, facilitating stakeholder communication.
• This method enhances the understanding and handling of missing data in complex, large-scale datasets.