A scalable framework for evaluating health language models.

Area of Science:

• Artificial Intelligence
• Computational Biology
• Health Informatics

Background:

• Large language models (LLMs) show promise in generating personalized health insights from patient data.
• Current LLM evaluation in healthcare relies on human experts, which is slow, costly, and prone to bias.
• Efficient and rigorous evaluation methods are needed to ensure the safety and accuracy of LLM health applications.

Purpose of the Study:

• To introduce Adaptive Precise Boolean rubrics, a novel framework for evaluating open-ended LLM responses.
• To streamline both human and automated evaluation processes for LLM-generated health content.
• To improve the scalability and cost-effectiveness of LLM assessment in complex medical domains.

Main Methods:

• Developed an evaluation framework using a minimal set of targeted Boolean questions to identify response gaps.
• Contrasted complex evaluation targets with precise, granular targets answerable with simple Boolean responses.
• Validated the framework in metabolic health, covering diabetes, cardiovascular disease, and obesity.

Main Results:

• Adaptive Precise Boolean rubrics achieved substantially higher inter-rater agreement compared to Likert scales for both expert and non-expert evaluators.
• The framework required approximately half the evaluation time of traditional Likert-based methods.
• Demonstrated improved efficiency and scalability, particularly with automated and non-expert assessments.

Conclusions:

• Adaptive Precise Boolean rubrics offer a more efficient and reliable method for evaluating LLMs in healthcare.
• The framework enhances scalability and cost-effectiveness, enabling broader adoption of LLM health applications.
• This approach facilitates more extensive and rigorous assessment of LLM-generated medical information.