Evaluating AI-assisted customer verification for synthetic nucleic acid screening

Area of Science:

• Biosecurity
• Synthetic Biology
• Artificial Intelligence

Background:

• Legitimacy screening verifies customers ordering synthetic nucleic acids to prevent misuse of synthetic biology.
• High costs associated with current screening practices hinder widespread adoption.
• AI tools offer a potential solution to streamline and reduce the cost of these essential security measures.

Purpose of the Study:

• To evaluate the effectiveness of large language models (LLMs) in facilitating legitimacy screening.
• To compare AI performance against human experts in terms of accuracy, source quality, and cost.
• To assess the cost-efficiency of AI in verification tasks for synthetic nucleic acid procurement.

Main Methods:

• Five LLMs were tested on five verification tasks using global life sciences researcher profiles.
• AI performance was benchmarked against an expert human baseline.
• Key metrics included flag accuracy, source quality, source fidelity, and cost per customer.

Main Results:

• The best-performing LLM, Gemini 2.5 Pro, achieved flag accuracy statistically indistinguishable from the human baseline (90.2% vs. 89.0%).
• Gemini 2.5 Pro outperformed or matched the human baseline in source quality and fidelity at approximately 10% of the cost ($1.18 vs. $14.04 per customer).
• AI-driven information gathering averaged $0.23 per customer, significantly cheaper than human screening.

Conclusions:

• AI assistance, particularly at the information-gathering stage, can significantly improve the efficiency and reduce the cost of legitimacy screening.
• LLMs show promise in enhancing biosecurity measures for synthetic nucleic acid providers.
• Human oversight remains crucial for final decisions in communication and order fulfillment.