An Artificial Intelligence Copilot for First Response Medicine

First responders in emergency medicine often work in austere conditions where expert consultation is limited, communication networks are unreliable, and rapid decision-making is essential. In these environments, procedural precision directly impacts patient outcomes. To address these challenges, we present an artificial intelligence (AI) Copilot for First Response Medicine, an augmented reality-based system that delivers real-time procedural guidance, decision support, and objective skill evaluation to frontline medics.

This is a feasibility and technical validation study. The system consists of 2 components: an augmented reality (AR) headset for multimodal user interaction and a Graphics Processing Unit (GPU)-equipped microprocessor for local execution of machine learning algorithms. This edge-based design provides sufficient computational power while ensuring offline operation, essential for low-resource and disconnected settings. The system integrates multiple models to analyze egocentric video streams from the headset and assist medics during procedures, including action recognition, action anticipation, visual question answering, hand tracking, and object detection, and can support skill assessment. Performance was benchmarked against expert-labeled ground truth using the Trauma THOMPSON dataset.

The MViTv2 model achieved the best performance for procedural guidance: 89.75% Top-5 accuracy for action recognition and 87.02% Top-5 for action anticipation. For context-aware clinical decision support, the Bootstrapping Language Image Pretraining (BLIP) model achieved 88.64% Visual Question Answering (VQA) accuracy. The system demonstrated robust 100% classification accuracy in differentiating expert and novice skill levels based on hand kinematic features, with average velocity and acceleration identified as the most influential factors.

Unlike AI-assisted surgical systems built for controlled hospital environments, the proposed AI Copilot is optimized for Role 1-2 field care, emphasizing autonomy, speed, and resilience. By offering proactive, real-time mentorship through AR, the system reduces cognitive load, enhances procedural accuracy, and provides an objective metric for continuous military medical training. This work bridges the gap between expert-level (Role 4) medical care and frontline responders and may improve the care in high-stakes and resource-limited scenarios.