Evaluating a large language model's ability to answer clinicians' requests for evidence summaries.

Area of Science:

• Medical Informatics
• Artificial Intelligence in Healthcare

Background:

• Clinical questions require timely and accurate evidence synthesis.
• Medical librarians provide gold-standard evidence summaries.
• Generative AI tools offer potential for assisting in evidence retrieval.

Purpose of the Study:

• To evaluate the performance of a generative AI tool (GPT-4) in answering clinical questions.
• To compare AI-generated responses with medical librarians' evidence syntheses.

Main Methods:

• Extracted clinical questions from an existing database.
• Developed a standardized prompt using the COSTAR framework.
• Utilized an internally managed chat tool (aiChat) with GPT-4 for response generation.
• Evaluated AI summaries against librarian-created gold standards.
• Verified a subset of AI-provided references.

Main Results:

• GPT-4 (aiChat) provided correct or partially correct answers for 99.5% of 216 clinical questions.
• No significant differences in ratings were observed across question categories.
• Only 37% of references cited by the AI tool were confirmed as non-fabricated.

Conclusions:

• Generative AI demonstrates promising performance in answering clinical questions.
• A significant portion of AI-generated references were unverifiable or fabricated.
• Further research is needed to understand AI integration into medical librarians' workflows.