Related Experiment Video
Updated: May 7, 2026
03:14
Augmenting Large Language Models via Vector Embeddings to Improve Domain-Specific Responsiveness
Published on: December 6, 2024
1.3K
Scoring Physician Risk Communication in Prostate Cancer Using Large Language Models
Guillermo Lopez-Garcia1, Dongfang Xu2, Michael Luu3
1Department of Computational Biomedicine, Cedars-Sinai Medical Center, Los Angeles, CA, USA, Guillermo.LopezGarcia@cshs.org.
Pacific Symposium on Biocomputing. Pacific Symposium on Biocomputing
|February 27, 2026
Summary
Large language models (LLMs) can now automatically score physician risk communication in prostate cancer care. This AI-driven approach offers a scalable solution for evaluating patient-doctor discussions.
Area of Science:
- Oncology
- Medical Informatics
- Artificial Intelligence
Background:
- Effective risk communication is crucial for shared decision-making in prostate cancer treatment.
- Current manual evaluation of physician communication is time-consuming and not scalable.
- Automating this assessment can improve the quality of patient-physician interactions.
Purpose of the Study:
- To develop and evaluate a framework using large language models (LLMs) for automated scoring of risk communication quality.
- To assess the performance of LLMs in evaluating physician communication of key concepts related to prostate cancer shared decision-making.
- To establish a scalable method for analyzing physician-patient communication in oncology.
Main Methods:
- A rubric-based framework was developed to score physician communication on five key concepts (prognosis, life expectancy, erectile dysfunction, incontinence, urinary symptoms).
- Transcripts from 20 clinical visits were annotated, with 487 sentences scored from 0 to 5 based on risk communication quality.
- The task was modeled as multiclass classification, evaluating finetuned transformer baselines and GPT-4o with rubric-based and chain-of-thought (CoT) prompting, including few-shot learning.
Main Results:
- The best performing approach combined rubric-based CoT prompting with few-shot learning.
- This method achieved micro-averaged F1 scores between 85.0% and 92.0% across different communication domains.
- The AI-driven approach outperformed supervised baselines and matched human inter-annotator agreement.
Conclusions:
- LLMs provide a scalable and accurate method for evaluating physician risk communication in prostate cancer consultations.
- This AI-driven framework can enhance the quality of shared decision-making by providing objective feedback on communication.
- The findings support the broader application of AI for assessing physician-patient communication in healthcare.
