Inter-tool analysis of a NIST dataset for assessing baseline nucleic acid sequence screening

Area of Science:

• Biosecurity
• Molecular Biology
• Bioinformatics

Background:

• Nucleic acid synthesis is a dual-use technology with significant benefits but also potential for misuse.
• Biosecurity protocols, including sequence screening, are crucial for responsible development and application.
• Existing DNA synthesis sequence screening tools vary in their performance and consistency.

Purpose of the Study:

• To assess the harmonization and consistency of DNA synthesis sequence screening tools.
• To evaluate the performance of various screening tools against a standardized dataset.
• To identify sources of disagreement in sequence screening to improve biosecurity measures.

Main Methods:

• NIST developed a test dataset based on current screening recommendations.
• Blinded datasets were distributed to sequence screening tool developers for evaluation.
• Performance metrics including sensitivity and accuracy were analyzed.

Main Results:

• A general agreement was observed between the screening tools and NIST's assignments.
• All tools demonstrated a baseline performance exceeding 95% sensitivity and 97% accuracy.
• Disagreements primarily involved single tools and stemmed from differing definitions of Sequences of Concern (SOCs) or methodological variations.

Conclusions:

• Current DNA synthesis sequence screening tools exhibit high baseline performance and general agreement.
• Variations in defining Sequences of Concern (SOCs) and screening methodologies contribute to screening disagreements.
• Further harmonization of screening criteria and methods is recommended to enhance biosecurity effectiveness.