Inter-tool Analysis of a NIST Dataset for Assessing Baseline Nucleic Acid Sequence Screening

Area of Science:

• Biosecurity
• Molecular Biology
• Bioinformatics

Background:

• Nucleic acid synthesis is a dual-use technology with significant benefits but also risks of misuse.
• Sequence screening is crucial for biosecurity protocols to identify potentially harmful synthetic nucleic acids (Sequences of Concern - SOCs).
• Existing screening tools vary in performance, necessitating evaluation of their consistency.

Purpose of the Study:

• To assess the harmonization and baseline performance of nucleic acid sequence screening tools.
• To evaluate the consistency of different tools in identifying Sequences of Concern (SOCs).

Main Methods:

• The National Institute of Standards and Technology (NIST) developed a test dataset based on current screening recommendations.
• Blinded datasets were distributed to developers of nucleic acid sequence screening tools for evaluation.

Main Results:

• A general agreement was observed between the screening tools and NIST-assigned labels.
• All tested tools demonstrated high baseline performance, with >95% sensitivity and >97% accuracy.
• Disagreements were primarily attributed to variations in defining SOCs and differences in screening methodologies.

Conclusions:

• Current nucleic acid sequence screening tools exhibit a high degree of consistency and accuracy.
• Differences in tool performance can be linked to variations in SOC definitions and algorithmic approaches.
• Further harmonization of SOC definitions may improve overall screening consistency.