Qualifying Certainty in Radiology Reports through Deep Learning-Based Natural Language Processing

Area of Science:

• Artificial Intelligence in Medicine
• Natural Language Processing
• Radiology Informatics

Background:

• Communication gaps persist between radiologists and referring physicians regarding diagnostic certainty.
• Current methods for assessing certainty in radiology reports are insufficient for precise communication.

Purpose of the Study:

• To explore deep learning-based bidirectional contextual language models for automatically assessing diagnostic certainty in radiology reports.
• To enhance the precision of communication between radiologists and referring physicians.

Main Methods:

• A dataset of 594 head MR imaging reports was randomly sampled.
• Three radiologists annotated 2352 sentences from the Impression section into four certainty categories.
• A natural language processing system using bidirectional encoder representations from transformers (BERT) was developed and validated.

Main Results:

• The biomedical variant (BioBERT) achieved the highest area under the curve (0.931) on validation data.
• All three BERT models demonstrated high macro-average specificity (93.13%-93.65%).
• BioBERT showed strong generalizability on heldout test data with an area under the curve of 0.93.

Conclusions:

• A deep transfer learning model can reliably assess the level of uncertainty communicated in radiology reports.
• AI-powered analysis of radiology reports can significantly improve diagnostic communication accuracy.