An analytical evaluation of contact tracing systems using real-world individual-level data

Contact tracing plays an important role in controlling contagious diseases. During the COVID-19 pandemic, new contact tracing systems with automation have been developed and adopted. Nevertheless, due to the privacy-protecting nature of automated contact tracing apps, evaluation based on real-world individual-level data has been scarce. Semi-automated contact tracing and the concurrent adoption of multiple contact tracing systems, though widely adopted, have also not received proper attention.

The objective of this study was to compare and evaluate concurrently adopted automated, manual, and semi-automated contact tracing systems using real-world individual-level data.

We collected and synthesized individual-level data from September 2020 to February 2021 at a university in the United States. We empirically analyzed and contrasted the case coverage, contact completeness, timeliness, and precision of different contact tracing systems. Privacy protection and contacts' actions were also investigated.

During our study time, 2415 individuals tested positive for COVID-19 on campus. Automated, semi-automated, and manual contact tracing systems had low overlap in case coverage (15%) and contact completeness (11% overlap in contacts that eventually became infected). Manual contact tracing achieved the highest case coverage (51% exclusive coverage) and contact identification precision (41% of identified contacts tested positive). Semi-automated and automated contact tracing was superior in timeliness, achieving a median of 1-day and 0-day test-to-report delay, respectively, much lower than the median of 6-day test-to-report delay in manual contact tracing. Of notified individuals, 93% took action to reduce COVID-19 transmission, and no one guessed/confirmed the case's identity merely through contact tracing notifications.

Automated, semi-automated, and manual contact tracing each have their superiorities and weaknesses. None of the systems is superior in all aspects. Concurrent adoption of automated, semi-automated, and manual contact tracing systems is desirable because they complement each other.