Using Large Language Models to Analyze Symptom Discussions and Recommendations in Clinical Encounters

Area of Science:

• Health Informatics
• Artificial Intelligence in Medicine
• Clinical Communication Research

Background:

• Analyzing patient-provider interactions is crucial for assessing care quality but is often hindered by time and methodological challenges.
• Large language models (LLMs) present a potential solution for analyzing these interactions, but their accuracy needs validation.
• Existing methods for evaluating clinical communication are resource-intensive, limiting their widespread application.

Purpose of the Study:

• To evaluate the accuracy and reliability of a large language model (LLM) in analyzing patient-provider communication within clinical encounter transcripts.
• To compare the coding performance of an LLM against human coders on key aspects of symptom discussions.
• To determine the feasibility of using LLMs as a research tool for analyzing patient-provider interactions.

Main Methods:

• A large language model (GPT-4) was used to code 236 potential symptom discussions from 92 cancer patient clinical transcripts.
• Human coders independently analyzed the same transcripts to identify symptom discussion, initiation, and recommendations.
• Cohen's kappa (κ) was calculated to measure interrater agreement between the LLM and human coders.

Main Results:

• The LLM demonstrated strong to moderate interrater reliability with human coders across all measures.
• Highest agreement was observed for symptom discussion (κ = 0.89), followed by initiation (κ = 0.82) and recommendations (κ = 0.78).
• Disagreements regarding recommendations occurred in 16% of cases, categorized into nine distinct types.

Conclusions:

• LLMs show comparable analytical abilities to humans in evaluating patient-provider interactions from clinical transcripts.
• The use of LLMs can significantly enhance the feasibility of analyzing patient-provider communication for research purposes.
• LLM-driven analysis holds potential for broader applications in assessing care quality, identifying inequities, and improving communication.