Enhancing Psychiatry Training Using an Agentic AI Simulated Consultation Tool: Prospective Cohort Study

Area of Science:

• Artificial Intelligence in Medical Education
• Psychiatric Training and Simulation
• Competency-Based Medical Education

Background:

• Canadian psychiatry residents require consultation competency assessment via the standardized assessment of a clinical encounter report (STACER).
• Current clinical settings offer limited opportunities for residents to practice consultation skills and receive feedback.

Purpose of the Study:

• To evaluate the technical feasibility of an agentic AI system for enhancing psychiatry residents' consultation competence.
• To assess the AI's capability in simulating patient encounters and providing structured feedback using a patient agent and rater agent.

Main Methods:

• A two-phase prospective cohort study of the STACER Agentic System, an LLM-based platform.
• Phase 1: Automated evaluation of the patient agent using a psychiatrist agent on 227 synthetic cases.
• Phase 2: User study with 14 participants (5 research team, 9 residents) evaluating simulated interviews and feedback.

Main Results:

• The patient agent showed high fidelity (91.07% behavioral, 95.45% symptom).
• Simulations were rated as psychiatrically plausible and diagnostically useful, though rapport building was moderate.
• The rater agent provided consistent feedback, with good-to-excellent interrater reliability at the section level for residents.

Conclusions:

• The STACER Agentic System is technically feasible for simulating psychiatric consultations and delivering feedback.
• Agentic AI shows promise for improving cognitive aspects of consultation skills and offering scalable practice opportunities.
• Further research is needed to evaluate educational effectiveness and integration into residency programs.