GenAI for Usability Testing of AR Apps: An Exploratory User Study

Augmented reality (AR) testing can be challenging, especially since the lack of regular user feedback is an issue in many domains. In this paper, we explore Generative AI (GenAI) as a method for testing AR usability without human users. We answer the question of whether GenAI can support usability testing of AR apps and, if so, how to utilise it effectively. We provided the model (here: GPT-5.1) with app key frames under two interaction protocols that differ in the timing of information delivery, illustrating how interaction affects the model's comprehension of the app. We evaluated the workflow by comparing the usability issues generated by the model with those reported by human users. For this, we developed a representative task and an AR app, and conducted 48 evaluations (24 human users and 24 independent GenAI trials). We analysed issue content in terms of emerging themes. Three AR experts rated the severity of the issues, and outputs were compared across sources. The results indicate that the model produces more unique issues, but also issues with a significantly lower mean severity score compared to issues produced by humans. The primary benefit of GenAI lies in its ability to generate well-articulated usability issues quickly. Beyond the studied task and application, the proposed workflow and interaction protocol offer a basis for transfer to other AR development contexts, but requires validation across additional applications and settings. GenAI is best positioned as a complementary tool for rapid, early feedback, rather than a replacement for human usability testing. A free copy of this paper and all supplemental materials are available at https://osf.io/x4a7c/.