Context-Aware Sentence Classification of Radiology Reports Using Synthetic Data: Development and Validation Study.

Area of Science:

• Artificial Intelligence in Radiology
• Natural Language Processing for Medical Data
• Synthetic Data Generation for Healthcare

Background:

• Automated structuring of radiology reports is crucial for data utilization and AI development.
• Manual annotation is labor-intensive, and real clinical data poses privacy risks, especially for non-English languages like Japanese.
• Synthetic data offers a privacy-preserving alternative, but its effectiveness for complex clinical nuances is unproven.

Purpose of the Study:

• To develop a context-aware sentence classification model for Japanese radiology reports using an entirely synthetic training pipeline.
• To eliminate reliance on real-world clinical data during model development.
• To evaluate the model's generalizability on diverse, multi-institutional real-world reports.

Main Methods:

• Generated 3,104 synthetic Japanese radiology reports using GPT-4.1 and annotated them with GPT-4.1-mini.
• Fine-tuned lightweight local LLMs (Qwen3, Llama 3.2) and Japanese text classification models (BERT, JMedRoBERTa, ModernBERT) using LoRA.
• Validated models on 280 real-world reports from J-MID, with labels confirmed by radiologists.

Main Results:

• Models achieved high performance on synthetic data (accuracy: 0.938-0.951, Macro F1: 0.924-0.940).
• Performance decreased on real-world data (accuracy: 0.783-0.813, Macro F1: 0.761-0.790), but positive predictive value for positive findings (PPV_1) remained high (e.g., 0.952).
• LLM-based approaches showed minor parsing errors (0.55%-7.48%) on the external dataset.

Conclusions:

• Developing context-aware sentence classification models for Japanese radiology reports using synthetic data is feasible.
• Models successfully captured key clinical terminology and patterns for identifying positive findings, as indicated by stable PPV_1.
• This synthetic data approach reduces annotation burden and privacy risks, supporting AI development in medical imaging.