Programming Chatbots Using Natural Language: Generating Cervical Spine MRI Impressions

Area of Science:

• Artificial Intelligence in Medicine
• Radiology Reporting Automation
• Natural Language Processing in Healthcare

Background:

• Machine learning, particularly large language models (LLMs), is increasingly explored in medicine.
• A gap exists in research on LLMs for generating specialized radiology impressions.
• Degenerative cervical spine MRI reports require accurate and clinically relevant impressions.

Purpose of the Study:

• To evaluate and compare the performance of multiple LLMs in generating radiology impressions for degenerative cervical spine MRI reports.
• To assess the diagnostic accuracy, stylistic accuracy, and redundancy of LLM-generated impressions.
• To identify the most effective LLM for this specific clinical application.

Main Methods:

• Comparative analysis of four LLMs: ChatGPT-3.5, GPT-4, Claude 2, Bard, and Llama 2.
• Few-shot learning approach using 50 synthetically generated MRI reports (10 examples).
• Evaluation based on diagnostic accuracy, stylistic accuracy, and redundancy metrics.

Main Results:

• Claude 2 maintained high performance consistently across 40 cases.
• GPT-4 required re-training to sustain performance; both Claude 2 and GPT-4 generated structured impressions.
• Claude 2's summarization capabilities offered an advantage in accuracy without continuous feedback; other LLMs performed poorly.

Conclusions:

• LLMs can automate radiology impression generation, offering a valuable clinical tool.
• Claude 2 shows significant potential for clinical implementation due to its consistent high performance.
• Further research is needed to optimize LLM performance and assess real-world clinical applicability.