An agentic AI system enhances clinical detection of immunotherapy toxicities: a multi-phase validation study

Area of Science:

• Oncology
• Artificial Intelligence
• Clinical Informatics

Background:

• Immune checkpoint inhibitors (ICIs) are crucial cancer therapies.
• Immune-related adverse events (irAEs) are common side effects of ICIs.
• Manual identification of irAEs from clinical notes is time-consuming and inconsistent.

Purpose of the Study:

• To develop and evaluate an agentic large language model (LLM) system for automated irAE extraction.
• To assess the system's performance in detecting presence, temporality, severity, attribution, and certainty of irAEs.
• To evaluate the impact of the AI system on efficiency and accuracy for clinical trial staff.

Main Methods:

• Development of an agentic LLM system for irAE extraction from clinical notes.
• Retrospective evaluation on 263 notes for detection and severity grading.
• Prospective silent deployment over three months on 884 notes.
• Randomized crossover study with 17 clinical trial staff members.

Main Results:

• Retrospective F1 score of 0.92 for detection and 0.66 for severity grading.
• Prospective detection F1 ranged from 0.72-0.79.
• Agentic AI reduced annotation time by 40% and increased complete-match accuracy.
• Improved inter-annotator agreement from 0.22-0.51 to 0.82-0.85.

Conclusions:

• Agentic AI systems can accurately and efficiently extract irAE information from clinical notes.
• AI-assisted irAE assessment enhances performance, consistency, and efficiency.
• This technology holds promise for improving patient safety and clinical trial management.