Enhancing Diagnostic Precision: Utilising a Large Language Model to Extract U Scores from Thyroid Sonography Reports

Area of Science:

• Artificial Intelligence in Medicine
• Radiology Reporting Automation
• Natural Language Processing Applications

Background:

• Thyroid ultrasound reports contain crucial U classification data.
• Manual extraction of U scores is time-consuming and prone to variability.
• Large Language Models (LLMs) offer potential for automating data extraction from clinical text.

Purpose of the Study:

• To evaluate ChatGPT-4's performance in automatically extracting U scores from thyroid ultrasound reports.
• To assess the accuracy of LLM-extracted U scores against human expert review.
• To identify challenges and potential benefits of using LLMs in radiology reporting.

Main Methods:

• Utilized ChatGPT-4 to extract U scores from 14,248 de-identified thyroid ultrasound reports.
• Provided LLM with U classification system guidelines.
• Compared LLM-extracted scores with initial clinician and refined expert-assigned U scores.

Main Results:

• Achieved 97.7% agreement between LLM-extracted and refined human U scores.
• Successfully identified the highest U score in 98.1% of multi-nodule reports.
• Discrepancies (2.5%) mainly due to ambiguous descriptions and report complexity.

Conclusions:

• LLMs demonstrate significant potential for accurate U score extraction from thyroid ultrasound reports.
• AI can enhance reporting consistency and reduce manual workload.
• Ethical considerations and workflow integration are crucial for clinical deployment.