Performance of an Open-Source Large Language Model in Extracting Information from Free-Text Radiology Reports.

Area of Science:

• Artificial Intelligence in Radiology
• Natural Language Processing for Medical Reports
• Machine Learning in Healthcare

Background:

• Emergency brain MRI reports contain critical information for patient care.
• Manual extraction of data from these reports is time-consuming and prone to error.
• Large Language Models (LLMs) show potential for automating information extraction from clinical text.

Purpose of the Study:

• To evaluate the performance of a local open-source LLM (Vicuna) in extracting data from emergency brain MRI reports.
• To assess the LLM's accuracy in identifying normal/abnormal findings, headache-causing vs. incidental abnormalities, and contrast medium use.

Main Methods:

• Retrospective review of 2398 emergency brain MRI reports from a French quaternary center (2022).
• Radiologists identified headache indications and categorized findings (normal/abnormal, headache-causing/incidental).
• The open-source LLM Vicuna performed the same information extraction tasks; performance was benchmarked against radiologist consensus.

Main Results:

• The LLM achieved high sensitivity and specificity for detecting headache context (98.0%/99.3%), contrast use (99.4%/98.6%), and report categorization (96.0%/98.9%).
• Performance for causal inference between MRI findings and headache was 88.2% sensitivity and 73% specificity.
• The LLM demonstrated excellent accuracy without requiring additional training on this specific dataset.

Conclusions:

• An open-source LLM can effectively extract information from free-text emergency brain MRI reports.
• The LLM shows high accuracy across various information extraction tasks, supporting its potential clinical utility.
• This demonstrates the feasibility of using readily available LLMs for automated analysis of radiology reports.