Creating reference data for early hospital outbreak detection algorithms from experts' ratings

Early outbreak detection algorithms can be useful tools to prevent healthcare-acquired infections in hospitals. However, their development and evaluation are hindered by the lack of available labelled data.

The aim of this study was to build a reference dataset for outbreak detection in hospitals using two different consensus approaches used to build this dataset.

25 Canadian and French experts were asked to review one-year time series of weekly incidence from different types of microorganisms, based on the data of a French university hospital. For each time series, experts also add access to additional surveillance data (locations and investigations). Each time series was submitted to three experts, whose role was to identify potential outbreak periods and rank their probability on a web platform. These rankings were summarized using two approaches: a majority vote wherein the most prevalent ranking was used for each week, and a hidden Markov model (HMM) in which the answers of the experts were used as observable variables that related to latent epidemiological states.

three experts reviewed a total of 36 times series i.e., 1899 surveillance weeks for 14 different types of microorganisms. Overall, the concordance between the two approaches was the highest for identifying high-ranking weeks. All rankings considered, 89 potential events were identified by majority vote and 96 by the HMM.

We constructed a reliable reference standard data set for the development, evaluation and comparison of algorithms for nosocomial outbreak detection within hospitals.