Synthetic data trained open-source language models are feasible alternatives to proprietary models for radiology...

Area of Science:

• Medical Informatics
• Artificial Intelligence in Radiology
• Natural Language Processing

Background:

• Free text dictations in radiology present challenges for structured data extraction.
• Proprietary large language models (LLMs) like GPT are powerful but raise privacy concerns.
• Open-source LLMs offer a potential alternative for medical data processing.

Purpose of the Study:

• To evaluate the feasibility of using synthetic data to fine-tune open-source LLMs for radiology text-to-structured data conversion.
• To compare the performance of fine-tuned open-source LLMs against GPT models.
• To assess the viability of open-source LLMs as a privacy-preserving alternative.

Main Methods:

• Generated 3000 synthetic thyroid nodule dictations for training.
• Fine-tuned six open-source LLMs (Starcoderbase, Mistral, Llama, Yi).
• Tested models on 50 MIMIC-III thyroid nodule dictations, comparing against GPT-3.5 and GPT-4 (0-shot, 1-shot, 5-shot).

Main Results:

• Yi-34B and GPT-4 5-shot achieved the highest performance with no significant difference.
• Several open-source models significantly outperformed GPT models.
• Models trained with synthetic data showed comparable performance to GPT models for structured text conversion.

Conclusions:

• Synthetic data is effective for fine-tuning open-source LLMs for radiology structured data extraction.
• Open-source LLMs demonstrate comparable performance to proprietary GPT models.
• Open LLMs present a privacy-preserving and viable alternative for clinical applications.