Clinical Risk Computation by Large Language Models Using Validated Risk Scores.

Area of Science:

• Artificial Intelligence in Medicine
• Clinical Informatics
• Natural Language Processing

Background:

• Large Language Models (LLMs) offer advanced natural language understanding for healthcare.
• Direct LLM risk prediction is unreliable due to bias and data complexity.
• Using LLMs to compute established clinical risk scores enhances validity and transparency.

Purpose of the Study:

• To evaluate the accuracy of public LLMs in calculating validated clinical risk scores.
• To compare the performance of GPT-4o-mini, DeepSeek v3, and Google Gemini 2.5 Flash.
• To assess LLM reliability for enhancing clinical workflows through interpretable score calculation.

Main Methods:

• Generated 100 diverse patient profiles as natural language clinical notes.
• Utilized LLMs (GPT-4o-mini, DeepSeek v3, Gemini 2.5 Flash) to extract data and compute five clinical risk scores.
• Compared LLM-computed scores against reference scores using accuracy, precision, recall, F1, and Pearson correlation.

Main Results:

• GPT-4o-mini and Gemini 2.5 Flash demonstrated near-perfect agreement with reference scores for most clinical risk assessments.
• DeepSeek v3 showed lower performance compared to GPT-4o-mini and Gemini 2.5 Flash.
• All evaluated LLMs encountered difficulties with the complex Framingham Risk Score calculation.

Conclusions:

• LLMs can accurately compute established clinical risk scores, offering a trustworthy alternative to direct AI risk prediction.
• GPT-4o-mini and Gemini 2.5 Flash show promise for integrating into clinical workflows for risk score calculation.
• Further development is needed to address LLM challenges with highly complex risk assessment formulas.