Application of Large Language Models in Complex Clinical Cases: Cross-Sectional Evaluation Study.

Area of Science:

• Medical Informatics
• Artificial Intelligence in Healthcare
• Clinical Decision Support Systems

Background:

• Large language models (LLMs) show promise for medical applications, but face challenges.
• Advancements in natural language processing (NLP) are driving LLM development.
• The integration of LLMs into clinical practice requires careful evaluation.

Purpose of the Study:

• To evaluate the efficiency, accuracy, and cost-effectiveness of LLMs in complex medical cases.
• To assess the potential of LLMs as clinical decision support tools.
• To compare LLM performance against traditional expert decision-making.

Main Methods:

• Evaluation of multiple LLMs using 80 complex cases from cardiothoracic surgery.
• Assessment of LLM decision-making time, accuracy (Likert scores), and cost.
• Comparison of LLM performance metrics against expert decision times.

Main Results:

• LLMs significantly outperformed experts in decision-making speed (under 4 minutes vs. 33.6 minutes).
• Deepseek-R1 and GPTo1 demonstrated superior accuracy and lower hallucination rates compared to other models.
• All evaluated LLMs offered substantially lower decision costs than multidisciplinary teams, with open-source models like Deepseek-R1 providing zero direct cost.

Conclusions:

• GPTo1 and Deepseek-R1 exhibit strong clinical potential due to enhanced efficiency, accuracy, and reduced costs.
• GPT4o and Kimi show moderate performance, suitable for broader clinical tasks.
• Further validation of LLaMa3 series and Gemini models in clinical decision-making is warranted.