Integrating Natural Language Processing With ChatGPT to Improve Quality of Artificial Intelligence Summaries of...

Patients with prostate cancer (PC) face complex treatment decisions and often struggle to recall information from consultations. Artificial intelligence(AI)-generated summaries have potential to improve recall, but summaries of key concepts from unstructured transcripts often lack focus and quality. We investigated whether natural language processing (NLP) preprocessing before AI summarization improves topic concordance and quality of risk information in AI summaries.

We audio-recorded and transcribed 42 PC consultations. Validated NLP models identified sentences about 5 content areas important for PC decision making: cancer prognosis (CP), life expectancy (LE), erectile dysfunction (ED), irritative urinary symptoms (IUS), and urinary incontinence (UI). ChatGPT-4.0 summarized raw transcripts and NLP-identified sentences across NLP probability thresholds from 50% to 90%. Summaries were reviewed for topic concordance, quality of risk information, and risk quantification. Logistic regression evaluated associations of these outcomes with NLP thresholds.

We analyzed 1,260 ChatGPT-4.0 summaries (6 per topic across 42 consultations). For each 10% increase in NLP probability threshold, the proportion of topic-related sentences in ChatGPT summaries increased by 18% for ED (incidence rate ratio [IRR], 1.18, 95% CI, 1.1 to 1.2), 16% for UI (IRR, 1.16, 95% CI, 1.1 to 1.1), 31% for IUS (IRR, 1.31; 95% CI, 1.2 to 1.3), 30% for LE (IRR, 1.30, 95% CI, 1.2 to 1.3), and 12% for CP (IRR, 1.12, 95% CI, 1.1 to 1.1). Odds of risk quantification increased for LE (odds ratio [OR], 1.39, 95% CI, 1.26 to 1.54, P < .001) and UI (OR, 1.18, 95% CI, 1.08 to 1.28, P < .001). Quality of risk information significantly increased for LE (0.16 points, 95% CI, 0.12 to 0.20) for each 10% increase in NLP threshold, with a clinically meaningful change (1 point) at a 60% threshold.

NLP preprocessing improves topic concordance, quality of risk information, and quantification in AI summaries of select content areas important for PC decision making, with potential to improve clinical care.