Ambiguous and Incomplete: Natural Language Processing Reveals Problematic Reporting Styles in Thyroid Ultrasound...

Area of Science:

• Medical Informatics
• Natural Language Processing
• Radiology Reporting

Background:

• Natural language processing (NLP) systems are crucial for converting unstructured clinical text into analyzable data.
• Thyroid ultrasound (US) reports contain vital information on nodules, but manual extraction is time-consuming and prone to errors.

Purpose of the Study:

• To evaluate the performance of NLP tools in extracting granular nodule details from thyroid US reports.
• To identify limitations in reporting language that affect NLP accuracy.

Main Methods:

• Iterative development of NLP tools using clinical Text Analysis and Knowledge Extraction System (cTAKES).
• Incorporation of nine nodule features for NLP extraction from 2007-2013 thyroid US reports.
• Evaluation of NLP precision, recall, and accuracy against a physician-adjudicated gold standard using separate test sets.

Main Results:

• NLP achieved 90% overall recall, with high recall for laterality (96%) and size (95%).
• Accuracy varied significantly across nodule characteristics, with lower scores for borders (40%), echogenicity (62%), and vascularity (65%).
• Reporting inconsistencies, such as ambiguous terms, accounted for 18% of NLP errors, impacting the extraction of features like contents and echogenicity.

Conclusions:

• NLP can automate critical information extraction from thyroid US reports.
• Ambiguous and incomplete reporting language significantly hinders NLP performance.
• Standardized or synoptic thyroid US reporting formats are recommended to enhance NLP accuracy and data utility.