Using set visualisation to find and explain patterns of missing values: a case study with NHS hospital episode...

Area of Science:

• Health Informatics
• Data Quality Management
• Data Visualisation

Background:

• Missing data is a prevalent issue in electronic health records (EHRs).
• Standard missing data checks are insufficient for understanding multifield missing data patterns.
• Advanced strategies require nuanced insights beyond simple counts.

Purpose of the Study:

• To develop and evaluate interactive set visualisation techniques for identifying multifield missing data patterns.
• To generate actionable insights from complex missing data in EHRs.
• To improve data quality assessment in large-scale health datasets.

Main Methods:

• Utilised interactive set visualisation techniques including bar charts, heatmaps, and histograms.
• Applied these methods to analyse anonymised admitted patient care health records from NHS hospitals.
• Processed over 16 million records across 86 fields, identifying multifield missing data patterns.

Main Results:

• Discovered unexpected missing data patterns in diagnosis, operation, and date fields.
• Identified sequential non-completion in diagnosis and operation fields, and missing dates for operations.
• Used information gain ratio and entropy to trace patterns to their origins within the data.

Conclusions:

• Set visualisation provides crucial insights into multifield missing data patterns within large EHR datasets.
• Revealed previously unknown rare and widespread data quality issues.
• Enabled pinpointing specific origins of data quality problems within hospital data.