Evaluation of the EsteR Toolkit for COVID-19 Decision Support: Sensitivity Analysis and Usability Study

Area of Science:

• Public Health
• Epidemiology
• Health Informatics

Background:

• Local health authorities managed COVID-19 cases in Germany.
• Contact tracing and monitoring were crucial for containment.
• The EsteR project developed statistical models to aid health authorities.

Purpose of the Study:

• Validate the EsteR toolkit's statistical models and web application.
• Assess model parameter stability (back end).
• Evaluate web application usability and applicability (front end).

Main Methods:

• Sensitivity analysis of 5 statistical models using literature-based COVID-19 parameters.
• Dissimilarity metrics and contour plots for model comparison.
• Cognitive walk-throughs and focus groups with 6 containment scouts for usability testing.

Main Results:

• Model sensitivity to parameter changes varied; stable parameter areas identified for single-use cases.
• Group use case stability depended on user input, lacking general stability.
• User feedback indicated a need for UI simplification and more guidance; the tool was deemed helpful, especially for new staff.

Conclusions:

• The EsteR toolkit was refined based on evaluation findings.
• Sensitivity analysis informed suitable model parameters and stability.
• Usability testing led to improvements in the web application's user-friendliness.