Closing the Loop: A Custom Artificial Intelligence Agent to Improve Detection of Radiologist Follow-Up...

Area of Science:

• Medical Informatics
• Artificial Intelligence in Healthcare
• Diagnostic Error Reduction

Background:

• Missed diagnostic opportunities are a critical subset of diagnostic errors, often stemming from failures in follow-up care management.
• Large health systems face amplified challenges in managing radiologist follow-up recommendations due to reliance on electronic health record (EHR) templates.
• Suboptimal use of EHR macros can lead to missed notifications and negatively impact patient care.

Purpose of the Study:

• To develop and implement a custom artificial intelligence (AI) agent to enhance the identification and management of follow-up recommendations in radiologist notes.
• To serve as an additional safety net within the digital health workflow to prevent missed diagnostic opportunities.
• To improve the integration of actionable findings for patient outreach and scheduling.

Main Methods:

• Developed a custom AI agent utilizing a pretrained large language model to review radiologist notes, specifically clinical impressions.
• The AI agent was designed to extract and standardize key follow-up details, integrating them into the digital health workflow.
• Model performance was evaluated on 10,000 radiologist notes and during 3 months of production use (over 120,000 studies).

Main Results:

• The AI agent achieved over 97% balanced accuracy in identifying notes requiring follow-up, flagging 6.18 times more cases than the existing system.
• Demonstrated over 94% accuracy in characterizing follow-up timing, recommended procedures, and underlying abnormalities.
• Enabled more reliable patient identification for follow-up and improved workflow integration for patient outreach.

Conclusions:

• Implementation of an AI agent as a safety net significantly improved the detection of missed diagnostic opportunities in radiology.
• The AI agent accurately extracted and standardized crucial details, aiding patient outreach and scheduling processes.
• This AI-driven approach enhances the reliability of follow-up care, optimizing patient outcomes in high-volume settings.