High-Performance Prompting for LLM Extraction of Compression Fracture Findings from Radiology Reports.

Area of Science:

• Radiology Informatics
• Natural Language Processing in Medicine
• Artificial Intelligence in Healthcare

Background:

• Manual extraction of spinal compression fractures from radiology reports is time-consuming and prone to errors.
• Large language models (LLMs) show potential for automating this process, but fine-tuning can be resource-intensive.
• Prompt-based strategies offer a less demanding alternative for optimizing LLM performance.

Purpose of the Study:

• To pioneer the use of Meta's Llama 3.1 with prompt-based strategies for automated extraction of spinal compression fractures.
• To evaluate the performance of different LLMs and prompting configurations without model training.
• To generate structured data from free-text radiology reports for improved clinical workflows.

Main Methods:

• Tested Meta's Llama 3.1 (70B and 8B) and Vicuna 13B models on 637 anonymized spine CT reports.
• Employed nine different prompting configurations, including radiologist-written and LLM-generated backgrounds, with and without few-shot examples.
• Compared model performance against manually generated ground truth annotations using F1 scores, ROC-AUC, and PR-AUC.

Main Results:

• The Llama 3.1 70B model achieved the highest F1 score of 0.91 with a radiologist-written background.
• Similar high performance (F1 score 0.86) was observed when the background was generated by another LLM.
• Prompt-based strategies demonstrated comparable ROC-AUC and PR-AUC performance, highlighting effectiveness without extensive training data.

Conclusions:

• Open-weights LLMs, specifically Llama 3.1, can effectively extract spinal compression fracture findings from free-text radiology reports using prompt-based techniques.
• This approach eliminates the need for resource-intensive model training and manual labeling.
• Automated extraction facilitates evidence-based care and empowers radiology workflows.