Automatic Annotation of Narrative Radiology Reports

Area of Science:

• Medical Informatics
• Natural Language Processing
• Machine Learning

Background:

• Electronic health records contain valuable narrative text data.
• Automated interpretation of radiology reports is crucial for clinical decision support systems.
• Developing query-capable hospital databases requires accurate labeling of free-form reports.

Purpose of the Study:

• To develop an automated method for labeling free-form radiology reports.
• To compare the performance of traditional machine learning models with deep learning approaches for text classification.
• To evaluate the effectiveness of different feature representations (Bag-of-Words vs. word embeddings) in classifying clinical conditions from radiology reports.

Main Methods:

• A dataset of 1295 knee radiology reports was manually labeled with 10 common clinical conditions.
• Two sets of text classification methods were compared: traditional models (Naive Bayes, Logistic Regression, Support Vector Machine, Random Forests) with Bag-of-Words features, and a Convolutional Neural Network (CNN) with word embeddings.
• Nested 10-fold cross-validation was used to evaluate performance metrics including accuracy, precision, recall, and F1 score.

Main Results:

• The Convolutional Neural Network (CNN) with word embeddings achieved the highest micro-averaged F1 score of 86.7%.
• CNN demonstrated strong performance for prevalent conditions like degenerative disease (95.9%), arthrosis (93.3%), and injury (89.2%).
• Traditional models (Logistic Regression, Random Forests, Support Vector Machine) performed comparably well (F1 scores 84.6%, 82.2%, 82.1%) but CNN underperformed on rare conditions due to data scarcity.

Conclusions:

• Automated labeling of radiology reports using CNN with semantic word representations is effective for clinical condition identification.
• Deep learning models show superior performance for well-represented classes, while traditional models offer robustness for underrepresented conditions.
• This automated approach facilitates the creation of query-capable report databases for enhanced clinical decision support.