A Dataset of Medical Questions Paired with Automatically Generated Answers and Evidence-supported References.

Area of Science:

• Artificial Intelligence
• Medical Informatics
• Natural Language Processing

Background:

• Large Language Models (LLMs) demonstrate advanced capabilities in medical question answering, improving answer quality.
• However, LLM-generated answers may lack factual accuracy, posing risks of misinformation.
• There is a critical need for datasets to assess and enhance the truthfulness of AI-generated medical information.

Purpose of the Study:

• Introduce the MedAESQA dataset for evaluating evidence-based medical question answering.
• Facilitate the development and fine-tuning of language models for factual attribution.
• Enable reliable detection of AI-generated answers not supported by scientific evidence.

Main Methods:

• Developed MedAESQA, a dataset with 40 aggregated, deidentified medical questions.
• Included 30 human and LLM-generated answers per question, with each statement linked to supporting scientific abstracts.
• Incorporated manual judgments on statement accuracy and scientific paper relevance.

Main Results:

• The MedAESQA dataset provides a benchmark for assessing factual accuracy in medical AI.
• It enables the evaluation of AI's ability to cite and link statements to reliable sources.
• Facilitates research into robust methods for detecting unsupported claims in medical AI.

Conclusions:

• MedAESQA is crucial for advancing trustworthy medical question answering systems.
• The dataset supports the creation of AI that generates factually attributable and evidence-based responses.
• Enhances the reliability and safety of LLM applications in healthcare.