Explainable XR: Understanding User Behaviors of XR Environments Using LLM-Assisted Analytics Framework

Area of Science:

• Human-Computer Interaction
• Immersive Analytics
• Extended Reality (XR)

Background:

• Existing XR analytics struggle with cross-virtuality transitions, multi-user collaboration, and multimodal data.
• Current frameworks lack a unified approach for analyzing diverse Extended Reality (XR) user behaviors.

Purpose of the Study:

• To introduce Explainable XR, a versatile framework for analyzing user behavior in diverse XR environments.
• To address limitations in current XR analytics by offering a virtuality-agnostic solution for data collection, analysis, and visualization.

Main Methods:

• Developed a novel User Action Descriptor (UAD) schema for capturing multimodal user actions, intents, and contexts.
• Created a platform-agnostic XR session recorder and a visual analytics interface.
• Integrated Large Language Models (LLMs) for assisted data interpretation and insight generation.

Main Results:

• Demonstrated Explainable XR's versatility through five use-case scenarios across individual and collaborative XR applications.
• Technical evaluations and user studies confirmed high usability and the delivery of multifaceted, actionable insights.
• The framework effectively handles multimodal data and cross-virtuality transitions in XR analytics.

Conclusions:

• Explainable XR provides a robust and usable solution for understanding user behavior in immersive environments.
• The framework facilitates deeper exploration and analysis of XR session data through LLM-assisted insights.
• Explainable XR enhances the study of human-computer interaction within the evolving landscape of Extended Reality.