Enhancing Disease Detection in Radiology Reports Through Fine-tuning Lightweight LLM on Weak Labels

Area of Science:

• Artificial Intelligence
• Medical Informatics
• Natural Language Processing

Background:

• Large language models (LLMs) show potential in medicine but face limitations like model size and data scarcity.
• Practical medical applications require specialized LLMs trained on relevant datasets.

Purpose of the Study:

• To investigate the effectiveness of fine-tuning a lightweight LLM (Llama 3.1-8B) using synthetic labels for medical tasks.
• To assess LLM performance with varying synthetic label quality.

Main Methods:

• Jointly fine-tuning Llama 3.1-8B on two tasks using combined instruction datasets.
• Utilizing synthetic labels generated by GPT-4o (high quality) and MIMIC-CXR (low quality).
• Evaluating performance on disease detection using micro F1 scores.

Main Results:

• Llama 3.1-8B achieved a micro F1 score of 0.91 on disease detection with high-quality synthetic labels.
• When fine-tuned with low-quality labels, the LLM surpassed its noisy teacher labels (0.67 vs. 0.63) after calibration.

Conclusions:

• Fine-tuning LLMs with synthetic labels is a viable strategy for medical domain specialization.
• Lightweight LLMs possess inherent capabilities that can be enhanced through synthetic data, even when labels are imperfect.