Where Your Eyes Go: How AI Output Design Impacts Reading Behavior

The purpose was to examine how variations in AI output design affect radiologists' performance in interpreting chest X-rays. Eight readers interpreted 80 COVID-19 chest images under five AI conditions in this retrospective study: no feedback, one-word summary, graph, heatmap, and heatmap + graph. Reader accuracy and eye-tracking data were analyzed to assess diagnostic performance and efficiency. Performance data were analyzed using a generalized mixed model nested for cases within readers assuming a binary distribution and with sandwich estimation; eye-tracking data were analyzed with analysis of variance. Baseline accuracy for detecting COVID-19 without AI was high and remained largely consistent across all AI designs. Fewer than 1% of decisions changed from correct to incorrect (true positive → false negative; true negative → false positive) with AI, while approximately 1% of decisions improved (false negative → true positive; false positive → true negative). More complex AI displays, such as the combined heatmap + graph, were associated with longer interpretation times and increased gaze shifts between the clinical image and AI outputs. Providing well-designed AI output can increase diagnosis accuracy and visual search of chest images. Simpler displays may support faster decision-making, whereas complex visualizations could impose additional cognitive demands to process the additional information. However, accuracy improvements likely outweigh modest increases in viewing time. Optimizing the presentation of AI information is essential to integrate human expertise effectively and create a synergistic human-AI partnership in clinical imaging, where the human remains the ultimate decision-maker.