LlymX: Multimodal LLM-Augmented XR for Context-Aware Information Access

Extended Reality (XR) and Large Language Models (LLMs) can support context-aware, hands-free information access in industrial environments. This paper presents LlymX, a prototype combining a Unity-based 3D environment with an LLM-driven backend leveraging retrieval-augmented generation (RAG) and vector databases. LlymX introduces an interaction-oriented XR-LLM pipeline in which spatial user actions, such as gestures and object selection, are grounded in semantic object representations and linked to retrieval from a structured knowledge base. Through speech, gestures, and text, users can identify objects and access contextually relevant information grounded in domain documentation. A user study comparing LlymX against menu-based and infographic-based XR interfaces shows that the LLM-based modality is significantly more usable (SUS) than static infographics and comparable to traditional menus. Task completion time and cognitive workload showed no statistically significant differences, though participants appreciated the flexibility of conversational interaction.