A framework for human-artificial intelligence co-learning for disease activity labeling using electronic health...

Area of Science:

• Artificial Intelligence in Healthcare
• Real-World Evidence Generation
• Clinical Informatics

Background:

• Generating real-world evidence (RWE) from electronic health records (EHR) is crucial but challenging due to the complexity and scalability issues of phenotyping from narrative clinical notes.
• Current methods struggle with standardization, auditing, and scaling the review of complex phenotypes, despite the potential of large language models (LLMs).
• Effective LLM applications require workflows that maintain evidence integrity, acknowledge uncertainty, and involve clinical experts in adjudication.

Purpose of the Study:

• To develop and evaluate the Synergistic Human-Agent REasoning system (SHARE), a novel framework for human-AI interaction.
• To enable accurate, robust, and reproducible phenotyping of complex outcomes from EHR data for RWE generation.
• To support scalable deployment of AI tools in clinical research while maintaining expert oversight.

Main Methods:

• Developed SHARE, a multi-stage human-AI co-learning framework utilizing rheumatoid arthritis (RA) disease activity as a use-case.
• Employed expert reviewers and a disease activity agent for note labeling, incorporating informative-note filtering, evidence extraction, and reasoning.
• Evaluated a budget-tiered configuration using different AI models for scalability, benchmarking against a high-effort configuration.

Main Results:

• Human-AI adjudication improved gold-standard labels, with the final agent's accuracy improving from a mean absolute error of 0.406 to 0.291.
• The SHARE framework achieved 92.1% accuracy in identifying ambiguous notes, matching expert designations.
• A budget-tiered configuration matched high-effort accuracy while reducing cost by 69% and compute time by 70% for large-scale cohort analysis.

Conclusions:

• The SHARE framework enhances the quality of gold-standard labels and facilitates accurate, standardized chart reviews at a scale unachievable through manual review alone.
• SHARE's human-AI co-learning approach identifies ambiguous cases for expert review, improving data reliability.
• The system's resource efficiency offers a transferable model for incorporating complex phenotypes into RWE studies, optimizing scalability and cost-effectiveness.