Use of Large Language Models to Predict Neuroimaging.

Area of Science:

• Medical AI
• Clinical Decision Support

Background:

• Large language models (LLMs) exhibit growing capabilities in the medical domain.
• Evaluating LLMs for specialized medical tasks is crucial for understanding their potential and limitations.

Purpose of the Study:

• To assess the ability of LLMs to predict optimal neuroradiologic imaging modalities based on clinical presentations.
• To compare the performance of LLMs against an experienced neuroradiologist in this task.

Main Methods:

• Two LLMs (ChatGPT and Glass AI) and a neuroradiologist evaluated 147 clinical scenarios.
• Responses were scored against the ACR Appropriateness Criteria.
• LLM outputs were analyzed for consistency and compared to expert performance.

Main Results:

• LLMs (ChatGPT: 1.75, Glass AI: 1.83) performed comparably but were significantly outperformed by the neuroradiologist (2.20).
• ChatGPT demonstrated greater inconsistency in its responses compared to Glass AI.
• No statistically significant difference was found between the two LLMs' overall performance.

Conclusions:

• LLMs are capable of suggesting appropriate neuroradiologic imaging but require further medical training for enhanced accuracy.
• Current LLMs do not surpass expert neuroradiologists in selecting imaging modalities.
• Continued development is necessary to improve LLM performance in complex medical applications.