Evaluating the Performance of Large Language Models on Palliative Care Test Questions: A Mixed Methods Study

Area of Science:

• Palliative Care Research
• Artificial Intelligence in Medicine
• Medical Education Technology

Background:

• Limited understanding of large language model (LLM) capabilities in palliative care (PC).
• Need to assess LLM performance on PC knowledge-based tasks.
• Evaluation of LLMs for answering PC questions and explaining rationale.

Purpose of the Study:

• To evaluate the performance of two large language models (LLMs) on palliative care (PC) knowledge-based questions.
• To assess the quality of LLM-generated explanations for their answer choices.
• To compare LLM-generated explanations with existing answer key explanations.

Main Methods:

• Two LLMs were prompted to answer 25 questions from the Fast Facts Quiz.
• LLMs provided rationale for their selected answer choices.
• Three PC educators rated and ranked LLM explanations against the Fast Facts Quiz answer key.
• Statistical analysis using linear fixed-effect models and ordinal logistic regression.

Main Results:

• Both LLMs achieved 96% accuracy in answering questions.
• LLM-generated explanations were rated higher by reviewers than the Fast Facts Quiz explanations.
• Reviewer feedback identified themes of perceived inaccuracies, clarity, educational value, linguistic style, and miscellaneous comments.

Conclusions:

• Large language models demonstrate high accuracy in answering palliative care questions.
• LLM-generated explanations are preferable to the Fast Facts Quiz answer key.
• LLMs show promise as tools for palliative care education and knowledge assessment.