Generative Artificial Intelligence for Medical Summarization in Prostate Cancer: Comparative Evaluation by Physicians...

Area of Science:

• Artificial Intelligence in Medicine
• Oncology Research
• Clinical Communication

Background:

• The rapid increase in scientific publications poses challenges for accessing up-to-date medical information.
• Language models (LMs) show promise for scientific content generation and summarization.
• LM performance in oncology needs evaluation from both professional and patient viewpoints.

Purpose of the Study:

• To evaluate the performance of four language models (Llama 3, Mistral Large 2, Gemma 2B, Consensus) in summarizing and translating prostate cancer abstracts.
• To assess the perceived quality of LM-generated summaries from physician and patient perspectives.

Main Methods:

• A prospective, survey-based pilot study was conducted.
• Four LMs summarized seven prostate cancer abstracts.
• Physicians and patients evaluated summaries based on accuracy, usefulness, organization, and comprehensibility using Likert scales.

Main Results:

• Physicians favored Consensus for completeness, accuracy, currency, organization, and usefulness.
• Patients rated Gemma 2B highest for conciseness and comprehensibility, and Consensus for organization.
• Llama 3 and Mistral Large 2 showed good overall performance but fewer strong positive ratings.

Conclusions:

• Perceived quality of LM summaries varied significantly between models and user groups (physicians vs. patients).
• Consensus was preferred by physicians, while Gemma 2B was favored by patients.
• Model selection for generative AI in oncology should align with specific clinical communication goals.