IDQuAD: Infectious disease question and answering dataset

Area of Science:

• Artificial Intelligence
• Infectious Diseases
• Natural Language Processing

Background:

• Large language models (LLMs) show promise across fields, but their application to infectious disease tasks is underdeveloped.
• Existing datasets lack the specificity needed for training and evaluating LLMs on infectious disease queries.

Purpose of the Study:

• To introduce the Infectious Disease Question and Answering Dataset (IDQuAD) for training and evaluating LLMs in infectious disease contexts.
• To address the gap in LLM application for infectious disease-specific question answering.

Main Methods:

• Constructed IDQuAD using medical papers, patents, and news.
• Employed novel methods like answer-before-question generation and counterfactual thinking to enhance QA pair quality.
• Fine-tuned the Mistral-7B model on IDQuAD and evaluated its performance in infectious disease QA tasks.

Main Results:

• The fine-tuned Mistral-7B model showed significant performance improvements, with an Exact Match (EM) score increasing from 28.49% to 65.47% in a one-shot setting.
• The fine-tuned model achieved the highest performance among all tested LLMs across various settings and metrics.
• Demonstrated the effectiveness of IDQuAD in enhancing LLM capabilities for infectious disease-related queries.

Conclusions:

• IDQuAD serves as a foundational dataset for advancing infectious disease research using LLMs.
• Fine-tuning LLMs on specialized datasets like IDQuAD is effective for improving performance on domain-specific tasks.
• This work paves the way for future dataset development and LLM refinement in infectious disease informatics.