Related Experiment Video
Updated: Jun 21, 2025
03:14
Augmenting Large Language Models via Vector Embeddings to Improve Domain-Specific Responsiveness
Published on: December 6, 2024
543
Outcome Prediction Using Multi-Modal Information: Integrating Large Language Model-Extracted Clinical Information and
Di Sun1, Lubomir Hadjiiski1, John Gormley1
1Department of Radiology, University of Michigan, Ann Arbor, MI 48109, USA.
Cancers
|July 13, 2024
Summary
Artificial intelligence (AI) large language models (LLMs) can extract clinical data to improve bladder cancer survival prediction after cystectomy. This multi-modal approach enhances prognostic accuracy using LLM-extracted information and imaging analysis.
Area of Science:
- Oncology
- Artificial Intelligence
- Medical Imaging
Background:
- Accurate survival prediction is crucial for bladder cancer patient management post-cystectomy.
- Artificial intelligence (AI) and large language models (LLMs) show potential in extracting clinical data and enhancing medical image analysis.
Purpose of the Study:
- To evaluate the efficacy of AI-LLMs in extracting clinical information for predicting five-year survival rates in bladder cancer patients post-radical cystectomy.
- To develop and assess a multi-modal predictive model integrating clinical, radiomics, and deep learning descriptors.
Main Methods:
- Retrospective analysis of medical records and CT urograms (CTUs) from 163 bladder cancer patients with post-surgery survival data.
- Extraction of clinical descriptors using five AI-LLMs (Dolly-v2, Vicuna-13b, Llama-2.0-13b, GPT-3.5, GPT-4.0) and manual extraction for comparison.
- Extraction of radiomics and deep learning descriptors from CTU images.
- Development of a multi-modal model (CRD) combining clinical, radiomics, and deep learning features.
Main Results:
- AI-LLMs achieved high extraction accuracies, ranging from 74% to 97%, with GPT-4.0 showing the highest performance (94%-97%).
- The CRD multi-modal model, utilizing LLM-extracted clinical data, demonstrated strong predictive performance (AUCs 0.81-0.88) comparable to models using manually extracted data (AUC 0.89).
- GPT-3.5 and GPT-4.0 based CRD models achieved AUCs of 0.85-0.88 and 0.87-0.88, respectively.
Conclusions:
- AI-LLMs can effectively extract clinical information from medical records for survival prediction in bladder cancer.
- Integrating LLM-derived clinical data with imaging analysis (radiomics, deep learning) improves the prediction of clinical outcomes post-cystectomy.
- This multi-modal approach offers a promising tool for enhancing prognostic accuracy and patient follow-up in bladder cancer care.
