Benchmarking Generative AI Tools for Interpretation of the WHO TB Mutation Catalogue

Area of Science:

• Bioinformatics
• Computational Biology
• Artificial Intelligence in Medicine

Background:

• The World Health Organization (WHO) 2023 Mutation Catalogue is essential for interpreting drug-resistant tuberculosis (TB) mutations but is complex and difficult to use.
• Challenges in accessing and interpreting the WHO Mutation Catalogue hinder effective TB control efforts.

Purpose of the Study:

• To evaluate the potential of generative artificial intelligence (AI) models for natural language interaction with the WHO Mutation Catalogue.
• To benchmark the performance of leading AI models in querying and interpreting TB mutation data.

Main Methods:

• A benchmarking study assessed four AI models: Google Gemini 2.5 Pro, OpenAI ChatGPT 4.1, Perplexity AI, and DeepSeek R1.
• Evaluations included general queries, mutation search/retrieval from the full catalogue and antibiotic-specific tables, and scoring novel mutations.
• Performance metrics comprised accuracy, completeness, clarity, source citation, and hallucination detection.

Main Results:

• Google Gemini 2.5 Pro demonstrated superior accuracy, completeness, and fewer hallucinations, particularly in general queries and large dataset searches.
• DeepSeek R1 excelled at applying grading rules and accuracy on focused datasets but had some hallucinations.
• ChatGPT 4.1 offered clarity but lacked citations; Perplexity AI had variable performance and more hallucinations.

Conclusions:

• Generative AI holds significant potential to enhance accessibility of complex knowledgebases like the WHO Mutation Catalogue.
• Rigorous benchmarking is crucial for developing reliable AI tools for TB mutation data.
• While not yet for clinical use, AI models like Gemini 2.5 Pro could form the basis of future agents to aid TB control efforts.