Evaluating Computer Vision and Augmented Reality Guidance to Optimize Weapon Training Safety and Performance

Mortars enhance the lethality of Warfighters and protect them during combat. Accurate use of weapon systems is critical for mission success and to minimize health hazards, such as blast overpressure exposure. Although safety guidance exists and helps prevent accidents and injuries, military training is dynamic, and human factors such as fatigue and elevated core body temperature can increase the risk of error. In collaboration across the Defense Center for Public Health-Aberdeen and Walter Reed Army Institutes of Research, and MITRE, the team developed and evaluated the Augmented Reality Guidance System (ARGS) to support Department of War (DoW) decision making, serve as a force multiplier to enhance training, and support public health (e.g., monitor blast exposures).

ARGS leverages emerging concepts in machine learning computer vision technology. It provides augmented reality, real-time, and after-action feedback to guide the actions of service members to mitigate unnecessary risks and support training and military operational medicine. ARGS uses the ZED stereo-camera, which is relatively low cost, waterproof, and intended for indoor and outdoor use, along with custom machine learning models for the DoW training use case. Through these models, ARGS evaluates the use of protective equipment by service members, the locations of individuals relative to weapon systems, and the use of the weapon systems. The team conducted 3 data collections at DoW mortar training events at 2 sites. The team implemented real-time facial blurring to preserve privacy and obtained feedback from instructors and unit leaders.

The ARGS models successfully identified helmet use, individual's heads above the level of the mortar muzzle (which increases blast overpressure exposure), and individuals located downrange. The first real-world data collection enhanced the existing computer vision models (80:20 train-test split), and the updated models resulted in perfect detection of 134 mortar rounds fired during that session. At the second data collection, the updated ARGS computer vision models were evaluated in real time. Across all 3 training events the models accurately identified 99% of 539 rounds fired from the 60 mm, 81 mm, and 120 mm mortars. In the first training event, there were 13 false positives (e.g., because of false detection during mortar tube cleaning). After assessment the team developed a model to detect mortar cleaning and was able to detect all 32 cleaning evolutions recorded at the sites with one false positive. Across the training events, ARGS identified an average of 5.88 individuals around the weapon systems and a high instance of personnel within 20 ft of the weapon system with their head above the level of the mortar muzzle (>90% of rounds fired), which could be an opportunity for changes to training to reduce exposure. In addition to safety metrics, ARGS output includes rate of fire data, which is a common performance measure.

ARGS shows promise for real-time support of range safety and after action review to enhance training effectiveness. ARGS identified and communicated safety information and deviations during training. This capability can help monitor and mitigate blast overpressure exposure and improve training to reduce injuries.