Domain Adaptation of a Large Language Model for Radiology Impression Generation - Multi-Reader Evaluation

To evaluate the clinical and practical trade-offs between domain adaptation and fine-tuning of large language models (LLMs) for radiology impression generation, with outcomes assessed through blinded expert radiologist review. In this retrospective institutional review board (IRB)-approved study 3,123,836 CT and MR reports were split into domain adaptation (80%), fine-tuning (16%), validation (2%), and test (2%) sets. Five LLM models were compared to ground truth: a base Llama-3-8B model (Meta) domain-adapted with mixed general, medical, and radiology text then fine-tuned using small (n = 1,280) and large (n = 499,814) number of report-impression pairs (DA + FTs, DA + FTl), an OpenBio-8B model with identical fine-tuning (Bio + FTs, Bio + FTl) and Gemini-1.5-Flash-002 used out-of-the-box. Four radiologists independently rated generated radiology impressions for abdominal CT and MR reports using six 5-point Likert items. Automated NLP metrics were also assessed. Domain adaptation improved performance in low-data regimes (mean score of DA + FTs 24.3 ± 2.8 vs. Bio + FTs 22.8 ± 3.5, P = 0.0004), but offered diminishing returns with large-scale fine-tuning (DA + FTl 24.9 ± 4.0 vs. Bio + FTl 25.0 ± 3.8, P = 0.89). Large-scale fine-tuned open-source models surpassed the commercial model in expert-rated quality and harmful output rates. Agreement among radiologists was low (ICC 0.22-0.49). Automated metrics showed weak correlation with human evaluation (ρ < 0.32). Domain adaptation improves LLM performance when labeled clinical data are scarce, but its incremental benefit diminishes as the fine-tuning dataset grows. Standard NLP metrics poorly reflect clinical relevance, underscoring the need for expert-informed evaluation. The findings help define the data-efficiency threshold for LLMs cost-effective deployment in clinical practice.