Automating the Observer OPTION-5 measure of shared decision making: Assessing validity by comparing large language...

Area of Science:

• Artificial Intelligence in Healthcare
• Clinical Communication Analysis
• Medical Decision-Making Research

Background:

• Observer-based measures of shared decision-making are resource-intensive due to reliance on human raters, limiting routine assessment.
• Generative artificial intelligence (AI) offers potential to enhance the speed and accuracy of evaluations while reducing human burden.

Purpose of the Study:

• To evaluate the performance of large language models (LLMs) from Gemini, GPT, and LLaMA families.
• To assess LLMs' ability to evaluate shared decision-making in early-stage breast cancer surgery consultations.

Main Methods:

• Compared LLM-generated scores with trained human raters using the Observer OPTION-5 measure.
• Analyzed 287 anonymized breast cancer consultation transcripts.
• Tested various prompts and evaluated LLMs' ability to distinguish high versus low scoring encounters.

Main Results:

• GPT-4o and Gemini-1.5-Pro-002 scores correlated with human ratings (Pearson r ≈ 0.6, p < 0.01).
• LLM performance approximated 75-80% of human inter-rater agreement (r = 0.77).
• LLMs successfully distinguished high- from low-scoring encounters (t > 10, p < 0.01).

Conclusions:

• LLMs can evaluate clinician-patient dialogue using existing measures for shared decision-making.
• Prompt development and fine-tuning are crucial for optimizing LLM performance.
• Future research should focus on generalizability and larger datasets to improve model performance.