Atherosclerosis prediction with large language models via prompt engineering

Atherosclerosis remains difficult to detect in its early stages due to its often asymptomatic nature. While artificial intelligence, particularly Large Language Models (LLMs), has shown significant promise for diagnostic support, optimal strategies for leveraging their capabilities in clinical risk prediction remain unclear. This study evaluated LLMs for atherosclerosis risk prediction using prompt engineering through mutual information-based feature selection and similarity-based case retrieval. Utilizing a large Japanese health checkup cohort (n = 117 698), with cardio-ankle vascular index (CAVI) as an atherosclerosis indicator, we evaluated Llama-3.1-Swallow-8B-Instruct-v0.1 performance across various configurations using F1 scores. We compared four case extraction methods, including Random Selection, Mahalanobis Distance, Cosine Similarity, Euclidean Distance, and feature selection from Zero-shot (no cases) to 4-shot, 6-shot, and 10-shot settings. The integration of mutual information-based feature selection with similarity-based case extraction improved F1 scores across all in-context learning settings. Cosine similarity-based selection achieved the highest F1 score in the 4-shot setting (0.715 ± 0.023) and 6-shot settings (0.719 ± 0.022), while Euclidean distance-based selection yielded the best F1 scores in the 10-shot (0.724 ± 0.017) settings. These combinations consistently outperformed the random selection baselines, which yielded significantly lower F1 scores (4-shot: 0.461 ± 0.017; 6-shot: 0.475 ± 0.023; 10-shot: 0.462 ± 0.029). Our findings demonstrate that combining feature selection with similarity-based case retrieval enhances the ability of LLMs to predict atherosclerosis risk. The substantial improvement in F1 scores over baseline methods suggests that optimized prompt engineering is crucial for clinical application of LLMs in cardiovascular risk assessment.