Data quality improvement of a multicenter clinical trial dataset

Area of Science:

• Medical Informatics
• Data Science
• Clinical Research Data Management

Background:

• Medical datasets often contain missing values, inconsistencies, and redundancies, hindering data mining and knowledge extraction.
• Effective data preprocessing is crucial for improving data quality and the reliability of insights derived from clinical trial data.

Purpose of the Study:

• To apply a five-step data preprocessing pipeline to a large multicenter clinical trial dataset.
• To evaluate the impact of each preprocessing step on data quality and patient classification accuracy.

Main Methods:

• Integrated data from multiple centers into a homogeneous dataset.
• Normalized variables, imputed missing values, and discretized/reduced the dataset.
• Evaluated data quality improvements using K-Nearest Neighbors (KNN) classifier for patient classification.

Main Results:

• The preprocessing pipeline resulted in a >20% increase in classification performance.
• Missing value imputation led to the most significant accuracy improvement.
• Discretization and feature selection reduced data volume without information loss.

Conclusions:

• The proposed data preprocessing pipeline effectively enhances the quality of medical datasets.
• Imputation of missing values is a critical step for improving predictive accuracy in clinical data.
• Data reduction techniques can be applied without compromising the integrity of the information for analysis.