Using Large Language Models to Identify Patient-Oncologist Communication Domains: A Feasibility Study

Area of Science:

• Medical Informatics
• Artificial Intelligence in Healthcare
• Oncology Communication

Background:

• The American Society of Clinical Oncology (ASCO) established patient-oncologist communication guidelines.
• Documenting these crucial conversations in medical records is ideal but chart review is inefficient.
• Large language models (LLMs) offer a computational approach to identify communication domains in clinical notes.

Purpose of the Study:

• To develop and validate an LLM-based approach for identifying communication domains within unstructured clinical notes.
• To compare LLM performance against gold-standard chart review for accuracy and efficiency.

Main Methods:

• Utilized a HIPAA-secure AI tool (GPT-4o) to develop an LLM prompt for identifying communication domains.
• Analyzed 134 clinical notes from 30 advanced cancer patients.
• Compared LLM identification of six communication domains against manual chart review using standard performance metrics and a hallucination index.

Main Results:

• LLM analysis demonstrated high accuracy, with sensitivity ranging from 0.43 to 1.0 and specificity from 0.32 to 0.99.
• The average hallucination index was low, indicating minimal false information generation.
• LLM abstraction took approximately 7 seconds per note, significantly faster than the 5-7 minutes required for chart review.

Conclusions:

• LLMs show significant potential for accurately identifying ASCO communication domains within clinical notes.
• This technology can streamline quality improvement efforts by providing rapid feedback to oncologists.
• Future applications may involve automated feedback generation for enhancing patient-oncologist communication.