Utilizing GPT-4o with context-based approaches for improved query of ACR RADS guidelines

Context-based large language models (LLM) with features like PDF integration and retrieval-augmented generation (RAG) may enhance accuracy on medical guideline queries.

200 representative questions were drafted across five major guidelines of the American College of Radiology (ACR) Reporting and Data Systems (RADS), with 40 questions each for CAD-RADS, BI-RADS, LI-RADS, PI-RADS, and Lung-RADS. Prompts were presented once to three GPT-4o-based models: without additional context, with PDF attachment of the guideline, and using web-based retrieval-augmented generation (RAG).

GPT-4o with PDF support answered 90% of questions correctly (180/200), outperforming both GPT-4o with RAG (83%, 165/200; P < 0.001) and GPT-4o without additional context (70%, 140/200; P < 0.001). The average response time was similar across all models: GPT-4o with PDF (13.6 ± 6.8 sec), GPT-4o with RAG (13.5 ± 3.5 sec), and GPT-4o without context (13.6 ± 6.3 sec; P = 0.97).

Incorporating contextual information such as guideline documents substantially improves the accuracy of GPT-4o in answering ACR RADS-related questions.