How green are large language models for radiology report labelling? Comparing human, rule-based and hybrid workflows.

Area of Science:

• Radiology and Medical Imaging
• Artificial Intelligence in Healthcare
• Computational Linguistics

Background:

• Quantitative data on the sustainable use of large language models (LLMs) in radiology is limited.
• Manual labelling of radiology reports is resource-intensive, involving significant time and cost.
• Developing efficient and sustainable data annotation workflows is crucial for AI implementation in healthcare.

Purpose of the Study:

• To quantify the resource footprint (time, cost, carbon emissions) of LLMs for labelling CT pulmonary embolism (PE) reports.
• To compare a hybrid rule-based-LLM workflow against manual labelling and LLM-only approaches.
• To identify sustainable deployment strategies for LLMs in routine radiology reporting.

Main Methods:

• A retrospective study involving 2923 structured CT reports.
• Four labelling workflows were evaluated: rule-based extractor (RBE), LLM-only, hybrid RBE-LLM, and manual labelling.
• Resource metrics (latency, cost, CO2 emissions) were measured for LLMs; labelling time was recorded by radiologists.

Main Results:

• Manual labelling required 32.8 hours and cost €0.42/report with 95.0% accuracy.
• LLM-only pipelines reduced time and cost but had lower accuracy (85.1%).
• The hybrid RBE-LLM workflow achieved the highest accuracy (98.5%) while significantly reducing time (0.97 hours), cost (€0.17), and CO2 emissions (0.12 kg) compared to LLM-only and manual methods.

Conclusions:

• LLM-only labelling offers reduced labor time and direct costs but compromises accuracy.
• A hybrid RBE-LLM approach sustains high accuracy while minimizing resource consumption.
• This hybrid workflow enables targeted LLM deployment, supporting sustainable data annotation in radiology.