Assessing the Reliability of SARS-CoV-2 Neutralization Studies That Use Post-Vaccination Sera

Assessing COVID-19 vaccine effectiveness against emerging SARS-CoV-2 variants is crucial for determining future vaccination strategies and other public health strategies. When clinical effectiveness data are unavailable, a common method of assessing vaccine performance is to utilize neutralization assays using post-vaccination sera. Neutralization studies are typically performed across a wide array of settings, populations and vaccination strategies, and using different methodologies. For any comparison and meta-analysis to be meaningful, the design and methodology of the studies used must at minimum address aspects that confer a certain degree of reliability and comparability. We identified and characterized three important categories in which studies differ (cohort details, assay details and data reporting details) and that can affect the overall reliability and/or usefulness of neutralization assay results. We define reliability as a measure of methodological accuracy, proper study setting concerning subjects, samples and viruses, and reporting quality. Each category comprises a set of several relevant key parameters. To each parameter, we assigned a possible impact (ranging from low to high) on overall study reliability depending on its potential to influence the results. We then developed a reliability assessment tool that assesses the aggregate reliability of a study across all parameters. The reliability assessment tool provides explicit selection criteria for inclusion of comparable studies in meta-analyses of neutralization activity of SARS-CoV-2 variants in post-vaccination sera and can also both guide the design of future neutralization studies and serve as a checklist for including important details on key parameters in publications.