Feasibility of Using the Privacy-preserving Large Language Model Vicuna for Labeling Radiology Reports.

Area of Science:

• Artificial Intelligence in Medical Imaging
• Natural Language Processing for Radiology

Background:

• Large language models (LLMs) like ChatGPT are unsuitable for radiology reports due to patient privacy concerns.
• Local deployment of LLMs is necessary to address privacy constraints in medical data.
• Vicuna-13B offers a potential alternative for local LLM applications in healthcare.

Purpose of the Study:

• To evaluate the feasibility of using the Vicuna-13B LLM for labeling chest radiography reports.
• To assess Vicuna-13B's performance in identifying 13 specific findings within radiography reports.
• To compare Vicuna-13B's labeling accuracy against established automated labelers and human radiologists.

Main Methods:

• Chest radiography reports from MIMIC-CXR and NIH datasets were retrospectively analyzed.
• Vicuna-13B was employed with single-step and multistep prompting strategies to identify 13 radiological findings.
• Agreement was assessed using Fleiss κ against CheXpert and CheXbert labelers, with AUC used for performance evaluation on a radiologist-annotated subset.

Main Results:

• Vicuna-13B, using a multistep prompting strategy, achieved moderate to substantial agreement (κ median 0.52–0.64) with CheXpert and CheXbert across both datasets.
• The model performed comparably to existing labelers (median AUC 0.84) for nine out of eleven findings.
• Inter-run agreement was assessed under randomized hyperparameter settings to ensure reliability.

Conclusions:

• The LLM Vicuna-13B demonstrates feasibility for labeling chest radiography reports, achieving moderate to substantial agreement with established methods.
• This study serves as a proof-of-concept for utilizing locally deployable LLMs in radiological report analysis.
• Vicuna-13B shows promise for privacy-preserving automated analysis of medical imaging reports.