An automated framework for assessing how well LLMs cite relevant medical references

Area of Science:

• Artificial Intelligence in Healthcare
• Medical Informatics
• Natural Language Processing

Background:

• Large language models (LLMs) are increasingly utilized for health-related inquiries.
• Ensuring LLM-generated health information is supported by credible references is critical.
• The degree to which LLM-cited sources actually support their claims is not well understood.

Purpose of the Study:

• To introduce SourceCheckup, an automated pipeline for evaluating source relevance and support in LLM responses.
• To assess the reliability of sources cited by popular LLMs for medical queries.
• To quantify the extent of source support for LLM-generated medical statements.

Main Methods:

• Developed an agent-based pipeline named SourceCheckup.
• Evaluated seven popular LLMs using a dataset of 800 medical questions.
• Analyzed 58,000 statement-source pairs from LLM responses to common medical queries.

Main Results:

• Between 50% and 90% of LLM responses were not fully supported or were contradicted by their cited sources.
• Even advanced models like GPT-4o with web search showed approximately 30% of statements unsupported.
• Independent medical professional assessments corroborated the findings of inadequate source support.

Conclusions:

• Current LLMs exhibit significant limitations in providing trustworthy medical references.
• A substantial portion of LLM-generated health information lacks adequate evidential support.
• Further development is needed to enhance the reliability of LLMs for health-related applications.