EpiCurveBench: Evaluating epidemic curve digitization.

Area of Science:

• Computational epidemiology
• Data science
• Medical informatics

Background:

• Accurate disease case counts over time are vital for training reliable disease forecasting models.
• Epidemic curve (epicurve) images, commonly used to display this data, are often in non-machine-readable formats.
• Manual digitization is time-consuming, and existing automated methods fail with complex, real-world epicurves.

Purpose of the Study:

• To address the limitations in automated epicurve data extraction.
• To create a comprehensive benchmark dataset for evaluating epicurve extraction methods.
• To introduce a novel evaluation metric that accurately assesses temporal data extraction.

Main Methods:

• Developed EpiCurveBench, a benchmark dataset of 100 manually curated and annotated epicurve images from diverse sources.
• Introduced EpiCurve Similarity (ECS), a new metric designed to evaluate the temporal structure and accuracy of extracted epicurve data.
• Evaluated state-of-the-art chart data extraction models on the EpiCurveBench dataset using the ECS metric.

Main Results:

• The EpiCurveBench dataset includes a wide variety of chart styles, from simple to complex.
• The novel EpiCurve Similarity (ECS) metric effectively captures temporal structures and handles variations in data length and completeness.
• The best-performing model achieved only 42.9% ECS on EpiCurveBench, indicating substantial room for improvement in automated epicurve extraction.

Conclusions:

• Existing automated methods for extracting data from epicurve images require significant improvement.
• The EpiCurveBench dataset and ECS metric provide a robust platform for advancing research in automated chart data extraction, particularly for epidemiological forecasting.
• This work facilitates the expansion of machine-readable epidemiological data, crucial for enhancing disease forecasting model accuracy.