Utilizing GPT-4o with context-based approaches for improved query of ACR RADS guidelines.

Area of Science:

• Artificial Intelligence in Medicine
• Medical Informatics
• Natural Language Processing

Background:

• Large language models (LLMs) show potential for medical question answering.
• Contextual information integration, such as PDF documents or retrieval-augmented generation (RAG), may improve LLM accuracy.
• American College of Radiology (ACR) Reporting and Data Systems (RADS) guidelines are crucial for standardized medical reporting.

Purpose of the Study:

• To evaluate the impact of contextual information on the accuracy of GPT-4o for answering queries based on ACR RADS guidelines.
• To compare the performance of GPT-4o with PDF integration versus RAG and no context.

Main Methods:

• 200 questions were developed across five ACR RADS guidelines (CAD-RADS, BI-RADS, LI-RADS, PI-RADS, Lung-RADS).
• Three GPT-4o models were tested: without context, with PDF guideline attachment, and with web-based RAG.
• Accuracy and response times were recorded for each model.

Main Results:

• GPT-4o with PDF support achieved the highest accuracy at 90% (180/200).
• GPT-4o with RAG achieved 83% accuracy (165/200), while GPT-4o without context achieved 70% accuracy (140/200).
• Response times were comparable across all tested conditions.

Conclusions:

• Integrating contextual information, specifically guideline documents via PDF, substantially enhances GPT-4o's accuracy for ACR RADS queries.
• PDF integration represents a highly effective method for improving LLM performance in specialized medical domains.