Transforming free-text radiology reports into structured reports using ChatGPT: A study on thyroid ultrasonography

Area of Science:

• Radiology
• Artificial Intelligence
• Medical Informatics

Background:

• Structured radiology reports are crucial for efficient data extraction and interprofessional collaboration.
• The application of large language models (LLMs) like ChatGPT in medical reporting is an emerging area of research.

Purpose of the Study:

• To evaluate the accuracy and reproducibility of ChatGPT (versions 3.5 and 4.0) in generating structured thyroid ultrasound reports based on ACR-TIRADS guidelines.
• To compare the performance of ChatGPT-3.5 and ChatGPT-4.0 in structuring these reports.

Main Methods:

• A retrospective analysis of 136 thyroid ultrasound reports (184 nodules) was conducted.
• ChatGPT-3.5 and ChatGPT-4.0 were utilized to structure the reports according to ACR-TIRADS criteria.
• Two radiologists assessed the quality, accuracy of nodule categorization, and management recommendations, with each report processed twice for consistency evaluation.

Main Results:

• ChatGPT-4.0 showed significantly higher accuracy in thyroid nodule categorization (69.3%) compared to ChatGPT-3.5 (34.5%).
• ChatGPT-4.0 provided more accurate and comprehensive management recommendations than ChatGPT-3.5.
• ChatGPT-4.0 demonstrated greater consistency in nodule categorization (ICC = 0.732) than ChatGPT-3.5 (ICC = 0.429).

Conclusions:

• ChatGPT holds significant potential for converting unstructured thyroid ultrasound reports into structured formats.
• ChatGPT-4.0 offers superior performance in accuracy and consistency for thyroid nodule categorization and management recommendations compared to ChatGPT-3.5.