Validation of a Difference-in-Differences Investigation Tool (DiD IT) for quantifying local outbreaks.

Area of Science:

• Public Health Surveillance
• Epidemiology
• Health Informatics

Background:

• Syndromic surveillance systems are crucial for monitoring public health threats.
• The Difference-in-Differences Investigation Tool (DiD IT) is a novel component of England's daily all-hazards syndromic surveillance.
• Accurate estimation of localized health events is essential for timely intervention.

Purpose of the Study:

• To validate the accuracy of the DiD IT in estimating small, localized increases in healthcare attendances.
• To assess the impact of different control settings within DiD IT on its performance.
• To evaluate the tool's effectiveness across various syndromic indicators and simulated scenarios.

Main Methods:

• The DiD IT was validated using synthetic data injects across ten syndromic indicators.
• Injects varied in size, duration, and spatial distribution, simulating real-world public health incidents.
• Control settings, including control location and time periods (e.g., washout periods, nearest neighbors), were systematically varied.

Main Results:

• DiD IT achieved a positive estimate in 99.8% of trials with a mean absolute error of 1.50.
• Confidence intervals for central estimates could not be generated in 42.5% of trials.
• The 95% confidence intervals included the actual inject count in only 62.8% of trials; performance varied with inject distribution and control settings.

Conclusions:

• The DiD IT demonstrates accuracy in assessing the impact of local public health incidents.
• Enhancements are required to improve the communication of estimate uncertainty to end-users.
• Further research should focus on refining uncertainty quantification and reporting within the DiD IT framework.