Building CAR-E: A Novel Artificial Intelligence Agent for Coaching Conversations

Coaching has many benefits in medical education, but large-scale implementation is constrained by faculty bandwidth, scheduling, cost, and power differentials. Artificial intelligence (AI), specifically large language models (LLMs), offers potential solutions to augment coaching, but limitations remain and refining performance is necessary to create valuable coaching conversations.

The authors created Coaching with AI-Reinforced Education (CAR-E), an AI coaching agent built using an LLM. Using layered architecture, CAR-E couples real-time speech input and output, retrieval-augmented generation (RAG) for evidence-based coaching, and dual memory system (short-term context and long-term history) to sustain longitudinal dialogue. An iterative design refined CAR-E with users who voluntarily engaged in coaching conversations and provided feedback. Transcripts and user feedback were analyzed to improve CAR-E.

During the pilot, 37 medical trainees and faculty engaged in coaching conversations. Transcripts showed diverse topics discussed and many strengths and opportunities for improvement. Users reported that CAR-E facilitated self-reflection, clarified goals, and broke complex problems into actionable steps. They also noted formulaic questioning that felt repetitive, superficial attempts to display empathy and difficulty moving conversations forward. Many users were frustrated by CAR-E's strict adherence to coaching competencies, which prevented it from giving advice or suggestions.

Unlike generic LLMs that default to broad, solution-oriented dialogue, CAR-E was designed for coaching in medical education. Its design incorporates evidence-based knowledge retrieval with curated coaching resources, structured memory to support longitudinal conversation, institutional oversight, and speech integration. These features exemplify responsible AI use in medical education. Despite this architecture, many improvements are planned to create reflective, growth-oriented conversations that augment coaching in medical education.