Utility of a Rule-Based Algorithm in the Assessment of Standardized Reporting in PI-RADS

Area of Science:

• Radiology
• Medical Informatics
• Natural Language Processing

Background:

• The Prostate Imaging Reporting & Data System (PI-RADS) is crucial for detecting clinically significant prostate cancer on multiparametric MRI (mpMRI).
• Standardized reporting is essential for consistent quality assessment and improvement in prostate cancer diagnosis.

Purpose of the Study:

• To develop and validate a rule-based algorithm using Regular Expression (RegEx) matching for automated categorization of prostate mpMRI reports.
• To assess the accuracy of the RegEx algorithm in classifying reports according to PI-RADS categories and identify areas for quality improvement.

Main Methods:

• Analysis of 6264 prostate mpMRI reports from Duke University Health System (2015-2021).
• Development of a RegEx algorithm to categorize reports into PI-RADS 1-5, Recurrent Disease, or "No Information Available."
• Validation of the algorithm against a blinded clinical team's review of 502 reports.

Main Results:

• The RegEx algorithm achieved 92.6% overall accuracy, 88.8% average precision, and 85.6% average recall.
• The algorithm demonstrated a strong F1 score of 0.871.
• Manual review identified 150 cases lacking numerical PI-RADS scores in the impression section, highlighting reporting inconsistencies.

Conclusions:

• Rule-based processing provides an accurate method for large-scale, automated extraction of PI-RADS scores from radiology reports.
• Natural language processing (NLP) approaches, like RegEx, can significantly enhance quality improvement initiatives in prostate mpMRI reporting.
• Automated analysis facilitates consistent PI-RADS categorization and identifies specific areas for improving reporting quality.