Context-Aware Sentence Classification of Radiology Reports Using Synthetic Data: Development and Validation Study

Area of Science:

• Medical Artificial Intelligence
• Natural Language Processing
• Radiology

Background:

• Automated structuring of radiology reports is crucial for data utilization and AI development.
• Manual annotation is labor-intensive, and real clinical data poses privacy risks, especially for non-English languages like Japanese.
• Synthetic data offers a privacy-preserving alternative, but its effectiveness for complex clinical nuances is unproven.

Purpose of the Study:

• To develop a context-aware sentence classification model for Japanese radiology reports using an entirely synthetic training pipeline.
• To evaluate the generalizability of the synthetic approach by validating on real-world, multi-institutional reports.

Main Methods:

• Generated 3104 synthetic Japanese radiology reports using GPT-4.1 and annotated them with GPT-4.1-mini.
• Fine-tuned lightweight local LLMs (Qwen3, Llama 3.2) and Japanese text classification models (BERT, JMedRoBERTa, ModernBERT-Ja).
• Validated models on 280 real-world reports from 7 institutions, with labels confirmed by radiologists.

Main Results:

• Models achieved high performance on synthetic data (accuracy: 0.938-0.951, macro F1: 0.924-0.940).
• Performance decreased on real-world data (accuracy: 0.783-0.813, macro F1: 0.761-0.790), but positive finding prediction (PPV_1) remained high (0.952-0.957).
• LLM-based approaches showed minor parsing errors (0.55%-7.48%) on external data.

Conclusions:

• Developing context-aware sentence classification models for Japanese radiology reports using synthetic data is feasible.
• The models successfully captured essential clinical patterns for identifying positive findings, despite performance variations.
• This synthetic data approach reduces manual annotation and privacy risks, enabling scalable dataset construction for medical AI.