Predicting Causes of Data Quality Issues in a Clinical Data Research Network.

Area of Science:

• Health Informatics
• Data Management
• Clinical Research Infrastructure

Background:

• Clinical Data Research Networks (CDRNs) face significant challenges in managing data quality.
• Manual investigation of data quality issues is time-consuming and inefficient.
• A large proportion of identified issues are not resolvable errors but inherent data characteristics or false alarms.

Purpose of the Study:

• To develop a predictive model for classifying the causes of data quality issues.
• To reduce the manual effort spent investigating non-resolvable data quality problems.
• To improve the efficiency of data quality management in CDRNs.

Main Methods:

• A machine learning classifier was trained using metadata from 10,281 real-world data quality issues.
• The classifier was designed to predict issue causes: extract-transform-load (ETL) code errors, inherent data characteristics, or false alarms.
• Performance was evaluated using the F1-measure.

Main Results:

• The developed classifier achieved an F1-measure of up to 90% in predicting the causes of data quality issues.
• This predictive capability can help prioritize investigations and reduce wasted effort.
• The methodology demonstrated effectiveness on the PEDSnet CDRN.

Conclusions:

• Predicting data quality issue causes is feasible and highly accurate.
• This approach can significantly streamline data quality management processes in CDRNs.
• The methodology is transferable to other CDRNs facing similar data quality bottlenecks.