Smart data-driven medical decisions through collective and individual anomaly detection in healthcare time series

Anomalies in healthcare refer to deviation from the norm of unusual or unexpected patterns or activities related to patients, diseases or medical centres. Detecting these anomalies is crucial for timely interventions and efficient decision-making, helping to identify issues like operational inefficiencies, fraud and emerging health complications.

This study presents a novel method for detecting both collective and individual anomalies in healthcare data through time series analysis using unsupervised machine learning. The dual-strategy approach leverages two methodologies: a 'practice centre-based approach' which monitors changes across different practice centres and a 'process-based approach' which focuses on identifying anomalies within individual centres. The former allows for early detection of systemic issues, while the latter highlights specific irregularities within a centre's operations.

The study utilised a dataset over 500,000 medical records from multiple GP practice centres in the UK collected between 2018-2023. Data are clustered using DBSCAN to identify collective anomalies from deviations from linear trends in consecutive two-month scatterplots. Individual anomalies are identified by examining the SOM-clustered time series of various medical processes within a specific practice centre, where graphs show deviation from the typical pattern.

Our approach addresses some challenges posed by the complexity and sensitivity of healthcare data by not requiring personal information. The method offers accurate visual representations making the data accessible and interpretable for non-technical users. Unlike traditional methods focusing solely on subsequence anomalies, our technique analyses the collective behaviour across multiple time series providing a more comprehensive perspective.

This study underscores the importance of integrating unsupervised anomaly detection with clinical expertise to ensure that statistically anomalous patterns align with clinical relevance. The dual-strategy clustering method holds significant potential for enabling timely interventions, proactively identifying potential crises, and ultimately contributing to better decision-making and operational efficiency within the healthcare sector.