Comprehensive Evaluation of Explanation Types in a Spaceflight-Relevant Human-Autonomy Teaming Task

ObjectiveThis study evaluates how explanation type in an explainable AI (XAI) human-autonomy teaming (HAT) task affects performance, workload, trust, situation awareness (SA), and preference in a dynamic, spaceflight-relevant simulator. Second, we introduce a holistic evaluation method for comparing XAI systems across multiple outcomes.BackgroundXAI aims to improve understanding, calibrate trust, and enhance performance of an HAT, but the impact of explanation type in realistic, high-taskload HAT settings remains underexplored.MethodParticipants ($N=31$) completed 18 trials in a dual-task simulator requiring manual rover driving while supervising an autonomous exploration agent. Participants received various combinations of global, contrastive, and deductive explanations for AI-generated routes, with incentives tied to performance.ResultsExplanation type significantly affected manual performance ($p=0.0003$), autonomy performance ($p<0.0001$), team performance ($p<0.0001$), workload ($p<0.0001$), trust ($p<0.0001$), and preference ($p=0.001$), but not SA ($p=0.41$). Participants preferred global and contrastive explanations, performing better with their preferred explanation ($p=0.049$).ConclusionExplanation type influences performance and perception in demanding HAT contexts. A standardized, multi-metric evaluation framework is essential for understanding tradeoffs in XAI design.ApplicationIn HAT tasks like space exploration where users must quickly make decisions with an AI teammate, designers must consider the explanation method for XAI explanations. Our human-centered evaluation found a contrastive + global explanation combination was the best in our HAT task across a range of performance and preference metrics.