Large Language Model-Based Evaluation of Medical Question Answering Systems: Algorithm Development and Case Study.

Area of Science:

• Artificial Intelligence in Healthcare
• Natural Language Processing
• Health Informatics

Background:

• Healthcare systems face resource constraints, limiting patient-provider interaction time.
• Conversational agents (CAs) offer potential solutions for patient information dissemination and query resolution.
• Ensuring CA accessibility requires understanding patient questions across various language and health literacy levels.

Purpose of the Study:

• To evaluate predefined medical content in CAs using Large Language Models (LLMs) across simulated patient populations with varying health literacy.
• To develop and apply a scalable evaluation framework for assessing CA performance.

Main Methods:

• Utilized LLMs to simulate patient populations with diverse health literacy levels.
• Developed a framework incorporating automated and semi-automated procedures for CA evaluation.
• Conducted a case study in mammography to assess LLM-driven CA evaluation.

Main Results:

• LLMs successfully simulated patient questions reflecting different health literacy levels.
• The accuracy of CA responses was found to be dependent on the simulated patient's health literacy.
• The mammography case study demonstrated the feasibility of the LLM-based evaluation approach.

Conclusions:

• The developed scalable framework facilitates the evaluation of domain-specific CAs for diverse patient populations.
• This framework supports the integration of CAs into clinical practice.
• Future work includes extending LLM applications to CAs with dynamic content and personalized information adaptation based on user health literacy.