Related Experiment Video
Updated: May 29, 2026
Augmenting Large Language Models via Vector Embeddings to Improve Domain-Specific Responsiveness
Published on: December 6, 2024
Evaluating large language models for structuring cardiology reports: a real-world clinical study on patient subtyping
Francesca Pescol1, Tommaso Mario Buonocore1, Valentina Tibollo2
1Department of Electrical, Computer and Biomedical Engineering, University of Pavia, Pavia, Italy.
Background:
Artificial Intelligence (AI) methods have emerged as useful tools for supporting patient recruitment in clinical trials (CTs). Despite several studies having recently proposed promising applications of Large Language Model (LLMs) for patient recruitment in CTs, their implementation in routine clinical practice remains limited.
Methods:
In this study, we present a comprehensive pipeline, developed and tested in a real-world clinical setting, to obtain highly detailed patient subtyping and eligibility assessment for specific CTs. Our solution leverages cardiological discharge letters, a rich yet underutilized source of patient data, to extract detailed structured clinical information through LLMs. Patient subtyping and eligibility assessment are performed through a rule-based approach, based on the extracted information, to maximize deterministic and interpretable outputs. We employed OpenAI's GPT-4.1 within the cloud-based service Microsoft Azure Machine Learning Studio, deployed in the hospital infrastructure. Validation was conducted on a sample of 100 discharge letters through exact-match comparison between the model's output and a ground-truth template, pre-populated by expert clinicians.
Results:
Our results confirm the feasibility and effectiveness of the proposed approach in real-world clinical scenarios. GPT-4.1 achieved high values of information extraction accuracy for most clinical variables (0.94 ± 0.08), resulting in a limited number of false negatives (FN) and false positives (FP) in both patient subtyping (0.12 and 0.13, respectively) and eligibility assessment. At the criterion-level, the proportion of FNs and FPs was below 3% for most criteria (13 and 11 of the 14 criteria examined, respectively).
Conclusion:
Overall, our study presents a notable step towards the integration of AI-driven approaches into real-world clinical practice for patient recruitment in CTs, highlighting both its practicality and effectiveness in meeting the stringent demands of healthcare settings.