Comprehensive Evaluation of Explanation Types in a Spaceflight-Relevant Human-Autonomy Teaming Task

Prior research has shown that artificial intelligence systems often lack transparency for human operators. This gap motivated investigations into how automated agents communicate their decision-making processes to users. No prior work had resolved how specific communication styles impact high-stakes environments like space missions. That uncertainty drove the need for realistic simulations involving dual-task demands. It was already known that trust calibration remains a persistent challenge in human-autonomy teaming. Designers frequently struggle to balance system performance with operator cognitive load. This study addresses the lack of standardized metrics for evaluating diverse explanation strategies. Understanding these dynamics is vital for future deep-space exploration missions.

The aim of this study is to evaluate how different explanation types in an Explainable AI system affect human-autonomy teaming performance. Researchers sought to determine the impact of these explanations on workload, trust, and situation awareness. The study addresses the challenge of maintaining effective collaboration in dynamic, high-taskload environments like space exploration. Motivation for this work stems from the need to calibrate human trust in autonomous agents. No prior research had fully explored how various communication styles influence operator efficiency in these specific settings. The investigators also intended to introduce a new, holistic evaluation method for comparing XAI systems. This framework is designed to help designers navigate the complex tradeoffs inherent in human-centered system development. Ultimately, the study provides evidence-based guidance for creating more effective AI teammates for future missions.

Review approach involved a controlled experiment with thirty-one participants completing eighteen distinct trials. The team employed a dual-task simulation requiring manual rover navigation alongside autonomous agent supervision. Researchers manipulated the presentation of global, contrastive, and deductive information provided by the AI. Performance incentives were integrated to ensure high levels of engagement throughout the testing process. The investigators gathered data on workload, trust, and situation awareness to assess the impact of these variables. A holistic evaluation framework was developed to synthesize these diverse metrics into a single comparative model. This approach allowed for the systematic analysis of how different explanation formats influence human behavior. The study design focused on capturing the complexities of high-taskload environments typical of future space missions.

Key findings from the literature reveal that explanation type significantly influences manual performance with a p-value of 0.0003. Autonomy performance also showed significant variance across conditions at p<0.0001. Team performance metrics followed this trend with a p-value of 0.0001. Workload levels were significantly impacted by the communication style at p<0.0001. Trust ratings similarly varied based on the explanation provided at p<0.0001. Participant preference for specific explanation types reached significance at p=0.001. Conversely, situation awareness did not show a statistically significant difference across conditions, yielding a p-value of 0.41. Participants performed better when receiving their preferred explanation type, which was confirmed at p=0.049.

The researchers propose that the style of information provided by an agent significantly alters operator efficiency and subjective experience. Synthesis and implications suggest that designers should prioritize specific combinations of feedback to optimize team output. The authors claim that a unified evaluation framework allows for a clearer understanding of design tradeoffs. Their findings indicate that contrastive and global formats are superior for user preference and task success. The study demonstrates that situation awareness does not necessarily change based on the provided explanation format. These results highlight the importance of human-centered testing in complex operational settings. The authors conclude that tailoring communication strategies is necessary for effective collaboration between humans and machines. Future design efforts should focus on these multi-metric approaches to improve overall system reliability.