Exploring the use of topological data analysis to automatically detect data quality faults

Area of Science:

• Data Science
• Computer Science
• Applied Mathematics

Background:

• Data quality issues are prevalent in structured and semi-structured data.
• Existing methods may be limited by data semantics, encoding format, or internal structure.

Purpose of the Study:

• To present an unsupervised, semantics-agnostic method for data quality analysis.
• To detect specific quality faults, such as records referring to the same entity.

Main Methods:

• Transforming data records into n-dimensional vectors to create a high-dimensional point cloud.
• Utilizing topological data analysis (TDA) to examine the point cloud's shape for anomalies.
• Employing a distance function for data transformation.

Main Results:

• The topological data analysis method effectively identifies high-dimensional anomalies indicative of data quality issues.
• The algorithm demonstrates robust accuracy across data of varying quality levels.
• The approach outperforms a baseline method, particularly for low-quality datasets.

Conclusions:

• Topological data analysis offers a powerful, unsupervised approach to data quality assessment.
• The method is versatile, applicable across different data formats and structures.
• This technique enhances the detection of subtle data inconsistencies, improving overall data integrity.