Using large language models to identify geriatric assessment domains in patients with advanced cancer: A feasibility...

In older adults with cancer, geriatric assessment (GA) can improve care quality. In-person assessment may not be feasible for all patients, and relevant information already exists in electronic health record (EHRs). However, chart review is time-consuming. Recently, large language models (LLMs) have demonstrated potential for automated abstraction and summarization tasks. The purpose of this study was to (1) develop an approach using LLMs to identify GA domains, and (2) demonstrate the potential of LLMs for creating GA summaries.

We extracted notes in the month following a poor prognosis for 30 randomly selected patients with cancer across seven clinical sites. We used a HIPAA-secure artificial intelligence tool to develop LLM prompts to identify and summarize domains. LLM was compared to chart review. A "hallucination score" was calculated for text included in the output. A "hallucination" is the term used to describe the production of false evidence.

Across 20 GA domains, note-level LLM analysis achieved sensitivity ranging from 0.44 to 1.0, specificity ranging 0.24-0.99, and accuracy ranging 0.49-0.98. Average hallucination index for documentation identified by the LLM was low. LLM frequently identified information that was documented in notes but missed by human experts. LLM-generated summaries included clinically-relevant information.

LLMs can abstract information relevant to GA domains with performance that is comparable to chart review, and in a fraction of the time. Although it would be ideal for all patients to receive in-person GAs, LLMs can identify relevant information when this is not feasible. The LLM exhibited low rate of producing false evidence, addressing one of the main concerns about clinical applications. LLM-generated GA summaries represent the first automated approach for this task.