Automated O-RADS Risk Stratification Using a Large Language Model Analysis of Narrative Ultrasound Reports

Area of Science:

• Artificial Intelligence in Medical Imaging
• Machine Learning for Diagnostic Support
• Natural Language Processing in Radiology

Background:

• Ovarian-Adnexal Reporting and Data System (O-RADS) standardizes ovarian lesion risk stratification.
• Manual O-RADS scoring is time-consuming and subject to inter-observer variability.
• Automated O-RADS scoring using large language models (LLMs) is investigated.

Purpose of the Study:

• To develop and evaluate an automated method for O-RADS scoring.
• To leverage LLMs for feature extraction from ultrasound reports.
• To improve the efficiency and consistency of ovarian cancer risk assessment.

Main Methods:

• A two-stage pipeline using the Lingshu LLM for feature extraction from narrative ultrasound reports.
• Identification of key diagnostic features by the LLM.
• Training and evaluation of machine learning algorithms (logistic regression, SVM, random forests) for O-RADS score prediction (1-5).

Main Results:

• The Lingshu LLM with logistic regression achieved an accuracy of 0.803 and AUROC of 0.948.
• This automated method outperformed the MedGemma model pipeline.
• The system demonstrated high performance in classifying ovarian lesions across a dataset of 513 cases.

Conclusions:

• A novel approach for automated O-RADS scoring using LLMs and machine learning was introduced.
• The method accurately stratifies ovarian cancer risk, enhancing clinical workflow efficiency.
• This automated system has the potential to reduce diagnostic variability and support radiologists' assessments.