Large language model integrations in cancer decision-making: a systematic review and meta-analysis.

Area of Science:

• Oncology
• Artificial Intelligence
• Medical Informatics

Background:

• Large Language Models (LLMs) are increasingly utilized to aid cancer patients and clinicians in medical decision-making processes.
• The integration and evaluation of LLMs within the field of oncology represent a rapidly evolving area of research.

Purpose of the Study:

• To systematically review the integration of LLMs into oncology practice.
• To analyze how researchers evaluate the performance and utility of LLMs in cancer care.

Main Methods:

• A comprehensive literature search was conducted across major databases (PubMed, Web of Science, Scopus, ACM Digital Library) up to May 2024.
• Included studies spanned 15 different cancer types, with 56 studies selected for analysis.
• Meta-analysis was employed to synthesize quantitative findings on LLM performance.

Main Results:

• LLMs are frequently used for summarizing, translating, and communicating clinical information in oncology.
• Average overall accuracy of LLMs was 76.2%, with diagnostic accuracy averaging a lower 67.4%.
• Evaluations predominantly used quantitative data and automated metrics, focusing on accuracy and appropriateness, while neglecting safety, harm, and clarity.

Conclusions:

• Current LLM performance and evaluation methodologies indicate limitations for clinical readiness in cancer decision-making.
• Key challenges include limited domain knowledge and the necessity for human oversight.
• Standardized evaluation frameworks and open datasets are crucial for enhancing LLM reliability in oncology.