Performance of Contradiction Rule Implementations in Health Data for Efficient Data Quality Assessments
Khalid O Yusuf1, Irina Chaplinskaya-Sobol1, Anne Schoneberg1
1Department of Medical Informatics, University Medical Center Göttingen, Germany.
Introduction:
Boolean rules are the building blocks for rule-based data quality assessment (DQA) in health research. While some DQA rules are generic, contradiction rules are guided by established facts supported by domain knowledge. A recent study reported performance degradation in infrastructure as DQA rules scale. Different implementation approaches can be used for DQA rules. In this study, we examine the performance of varied DQA rule implementations for contradictory dependencies in data items for cardiovascular disease assessment and propose an optimization method that integrates the strengths of different approaches.
Methods:
We implemented three Boolean rule implementations considered for contradiction assessment of 12 cardiovascular disease items used in the cross-sectoral platform of the national COVID-19 cohort: 1) raw domain rule-set joined using the Boolean-OR operator; 2) two minimal Boolean rules derived from the twelve raw domain rule-set through rule reduction; and 3) atomic Boolean rules representing each rule in the raw domain rule-set. The implementations are examined on speed of execution and memory utilization on the original dataset of about 2000 subjects amplified by factors of 2.5, 5, 10, 50, and 100. A two-step approach is adopted to integrate the implementation of the fastest and atomic contradiction rules.
Results:
The raw domain rule-set (1) was more than 100 times faster than the atomic rules (3) and 9 times faster than the minimal Boolean rules (2) with the largest employed dataset. It requires about 3 times more memory than the other implementations. All implementations show linear dependency on the dataset size, except for minimal Boolean rules (2) with a slower slope in memory utilization. Two-step rule processing reduced the speed gap between raw rule-set (1) and atomic rules (3) from 100 times faster to just 3 times in the unified implementation.
Discussion:
Only atomic rules (3) support detailed and traceable results for DQA, required for further inspection of the contradictions. A combined rule processing can bridge the speed gap between raw rule-set and atomic rules by executing the fastest rules on entire dataset and atomic rules only on the fraction of data with contradictions, allowing for fast but detailed DQA.
Related Concept Videos
01:11Purpose of Health Records I
Here's a breakdown of how health records serve these purposes:
01:03Data Validation
Nursing assessment guides are generally based on holistic models rather than medical...
01:30Health Information Technology and Healthcare Information System
Health Information Technology, commonly called HIT, integrates advanced information systems and technology in healthcare settings. Its primary functions include:
01:13Multiple Comparison Tests
It would be easy to compare two samples using a significance alpha level of 0.05. In other words, there is only one sample pair to be compared. However, it would be difficult to identify a significantly different sample if the number...
01:19Quality Assurance
01:23Types of Biopharmaceutical Studies: Controlled and Non-Controlled Approaches
Non-controlled studies, commonly employed for initial exploration, lack a control group, rendering them susceptible to biases and external influences. In contrast,...