Assessing Laterality Errors in Radiology: Comparing Generative Artificial Intelligence and Natural Language...

Area of Science:

• Radiology
• Artificial Intelligence
• Medical Informatics

Background:

• Laterality errors in radiology reports can lead to misdiagnosis and patient harm.
• Accurate identification of laterality is crucial for diagnostic integrity.
• Current natural language processing (NLP) tools have limitations in detecting these errors.

Purpose of the Study:

• To compare the performance of generative AI (Augmented Transformer Assisted Radiology Intelligence - ATARI) and NLP tools in identifying laterality errors in radiology reports and images.
• To assess the accuracy of both tools in distinguishing true reporting errors from NLP-detected false positives.

Main Methods:

• An NLP tool flagged radiology reports for potential laterality errors.
• A radiologist validated these flags as true reporting errors or NLP false positives.
• Generative AI (ATARI) was applied to a subset of reports with true and false positive errors for accuracy assessment.
• Both text-only and combined text-and-image queries were used with ATARI.

Main Results:

• Of 898 NLP-flagged errors, 64% were NLP errors (false positives) and 36% were true reporting errors.
• ATARI's text query achieved 97.4% accuracy in identifying absence of laterality mismatch (NLP false positives).
• Combined text and image queries with ATARI reached 98.3% accuracy in identifying laterality errors.

Conclusions:

• Generative AI (ATARI) demonstrated superior performance over NLP in detecting laterality errors in radiology.
• ATARI's ability to incorporate image analysis enhances its accuracy for laterality determination.
• Further refinement of ATARI's text query function is needed for complex reports.