Implementing a Resource-Light and Low-Code Large Language Model System for Information Extraction from Mammography...

Area of Science:

• Artificial Intelligence in Medical Imaging
• Natural Language Processing in Radiology

Background:

• Large language models (LLMs) show promise for extracting data from free-text radiology reports.
• Current research primarily uses LLMs via APIs, limiting accessibility and local deployment.

Purpose of the Study:

• To evaluate the feasibility of using open-source LLMs on limited local hardware for data extraction from mammography reports.
• To assess the impact of different prompting strategies on extraction performance.

Main Methods:

• Defined 79 common data elements (CDEs) for mammography reports.
• Used five open-source LLMs deployable on a single GPU for data extraction.
• Compared five prompt approaches, including default, chain-of-thought, and few-shot prompting.
• Analyzed performance using accuracy, micro-recall, micro-F1, and certainty thresholds.

Main Results:

• High inter-rater agreement (Cohen's kappa 0.83) established ground truth.
• Default LLM prompts achieved 59.2-72.9% accuracy.
• Task-specific prompt adaptation improved accuracy to 64.7-85.3%.
• Certainty thresholds boosted accuracy over 90% but reduced coverage below 50%.

Conclusions:

• Open-source LLMs are effective for extracting information from mammography reports using limited computational resources.
• Model selection and prompt engineering are critical for optimal performance.
• A CDE-based framework enhances data extraction clarity and structure.