Assessment of Zero-Shot Large Language Model (LLM) Assisted Clinical Trial Matching Processes: A Metastatic Cancer
Yingjie Weng1, Himani Yalamaddi1, Danning Fu1
1Quantitative Sciences Unit, Stanford University School of Medicine, Palo Alto, CA.
Introduction:
For oncology patients with limited treatment options, clinical trials may be a critical lifesaving pathway. Identifying relevant trials, however, is a time-consuming and difficult task. Several patient-trial matching processes incorporating large language models (LLMs) have been proposed to alleviate the burden on patients and oncologists. We aim to explore the benefits and practical challenges of zero-shot LLM-assisted trial matching processes by analyzing the results for a single pancreatic cancer patient.
Materials And Methods:
The results of a simple zero-shot LLM-assisted clinical trial matching process for our patient were compared to those of a "human benchmark," which was developed manually by two of the authors interfacing directly with ClinicalTrials.gov. Performance metrics - sensitivity, specificity, precision, and accuracy - were calculated. In addition, a qualitative content analysis (QCA) of LLM reasoning text was done to identify patterns in "errors," which we define as a human-LLM discrepancy in final patient eligibility. Implications and severity of errors are discussed.
Results:
The zero-shot LLM-assisted process returned potential trials with a sensitivity, specificity, and precision of 81.1%, 89.3%, and 86.5% respectively compared to the human benchmark. Qualitative error analyses revealed that about 73% of errors could potentially be alleviated with improved prompting and information access. Overall performance seemed comparable to that of human reviewers.
Conclusion:
The results from this preliminary real-world case study provide additional evidence to the literature in support of the integration of LLMs in clinical trial matching to provide benefit to patients with metastatic cancer with limited options.
