LlymX: Multimodal LLM-Augmented XR for Context-Aware Information Access.

Area of Science:

• Human-Computer Interaction
• Industrial Informatics
• Artificial Intelligence

Background:

• Extended Reality (XR) and Large Language Models (LLMs) offer potential for context-aware, hands-free information access in industrial settings.
• Current XR interfaces often lack dynamic, context-specific information retrieval capabilities.

Purpose of the Study:

• To present LlymX, a prototype system integrating XR with LLMs for enhanced industrial information access.
• To evaluate the usability and effectiveness of an LLM-driven XR interface compared to traditional methods.

Main Methods:

• Developed LlymX, a prototype combining a Unity-based 3D environment with an LLM backend using retrieval-augmented generation (RAG) and vector databases.
• Implemented an interaction-oriented XR-LLM pipeline grounding spatial user actions with semantic object representations.
• Conducted a user study comparing LlymX against menu-based and infographic-based XR interfaces.

Main Results:

• The LLM-based LlymX interface demonstrated significantly higher usability (SUS score) than static infographics.
• Usability was comparable to traditional menu-based XR interfaces.
• No significant differences were found in task completion time or cognitive workload.

Conclusions:

• LLM-driven XR interfaces provide a highly usable and flexible modality for accessing industrial information.
• LlymX demonstrates the potential of integrating LLMs with XR for context-aware, hands-free industrial applications.
• Users appreciate the conversational interaction flexibility offered by LLM-powered XR systems.