Cross-Institutional Evaluation of Large Language Models for Radiology Diagnosis Extraction: A Prompt-Engineering...

Area of Science:

• Artificial Intelligence in Medical Imaging
• Natural Language Processing for Healthcare

Background:

• Large language models (LLMs) show potential for automating radiology report analysis.
• Accurate annotation of radiology reports is crucial for clinical decision-making and research.

Purpose of the Study:

• To evaluate the effectiveness of a human-optimized prompt for labeling radiology reports using LLMs across multiple institutions.
• To assess the consistency and accuracy of LLM-based annotation in a multi-institutional setting.

Main Methods:

• Six institutions collected 500 radiology reports across 5 categories.
• A standardized Python script executed a common LLM with a human-optimized prompt locally at each site.
• LLM predictions were compared against investigator-provided reference labels to calculate accuracy.

Main Results:

• The human-optimized prompt demonstrated high consistency across different institutions and pathologies.
• Significant agreement was observed between LLM outputs and investigator labels.
• Llama 3.1 70b achieved the highest performance in identifying specified findings, showing robust adaptability across sites.

Conclusions:

• Optimized prompt engineering with LLMs is a viable approach for cross-institutional radiology report labeling.
• The method is straightforward, accurate, and adaptable to variations in report structure and institutional practices.
• Future research should focus on model robustness and prompt generalizability for diverse report structures.