Factors affecting the labelling accuracy of brain MRI studies relevant for deep learning abnormality detection

Area of Science:

• Artificial Intelligence
• Medical Imaging
• Natural Language Processing

Background:

• Deep learning computer vision classification requires extensive datasets for training.
• Natural Language Processing (NLP) offers automated dataset labeling as a potential solution.
• The efficacy of NLP for medical dataset labeling, particularly in neuroradiology, requires validation.

Purpose of the Study:

• To develop and validate a deep learning-based NLP classifier for neuroradiology MRI reports.
• To assess the accuracy of NLP labeling for binary (normal vs. abnormal) and multi-class disease categories.
• To investigate the impact of labeller expertise on NLP model performance.

Main Methods:

• Manual labeling of over 5,000 head MRI reports by expert radiologists.
• Development of a deep learning-based NLP classifier.
• Evaluation of model accuracy using binary and multi-class labels with varying MRI sequences.
• Comparison of model performance based on labeller expertise (expert vs. non-expert).

Main Results:

• High accuracy achieved for binary classification (normal vs. abnormal) using limited MRI sequences (T2-weighted, diffusion-weighted imaging).
• Accuracy for multi-class disease categorization was variable and category-dependent.
• Model performance was significantly influenced by the expertise of the original data labeller, with expert-labeled data yielding better results.

Conclusions:

• NLP shows promise for automating neuroradiology report labeling, especially for binary classification.
• The specificity of labels and the expertise of data annotators are critical factors for successful NLP implementation in medical AI.
• Further research is needed to optimize NLP for complex, multi-class medical data labeling.