A comprehensive framework for de-duplication: Acute kidney failure (AKF) case study

Objectives: Addressing data duplication is one of the most important issues in electronic health record (EHR) processing since the nature of data collection in the field. It does not only affect the data quality in healthcare management, but also the reliability in the downstream analyses. In this paper, we propose a comprehensive data de-duplication framework tailored for medical databases to tackle data duplication for a kidney disease identification, Acute Kidney Failure (AKF). Methods: The proposed work begins with the data joining from various sources, basic data de-duplication which automatically removes the dirty texts, medical note-event extraction since the data could be sources for further de-duplication, NLP data de-duplication based on a pre-trained model, data mapping for integration, unrelated data and outlier elimination, and eventually data imputation by a clustered based imputer. Results: We illustrated our de-duplication framework on MIMIC-III database both on the de-duplication task and the classification task based on AKF. The experiments demonstrated that the proposed work could achieve up to 99.59% accuracy or 23% higher than the traditional method and could achieve a high classification accuracy at 86 % and the F1-score at 0.87, which outperformed the traditional method, and the original dataset without any modification. Conclusion: These results demonstrated that the framework can potentially address the data duplication issue in healthcare effectively.