Semantic Scene Graphs for Creating a Localization-Ready Internet of Things.

Area of Science:

• Human-Computer Interaction
• Internet of Things (IoT)
• Augmented Reality (AR)

Background:

• Fragmented user experience in controlling Internet of Things (IoT) devices due to multiple apps and remotes.
• Augmented Reality (AR) offers potential for superior device control via spatially aligned virtual interfaces.
• Accurate physical localization of IoT devices is a prerequisite for AR-based control.

Purpose of the Study:

• Introduce LORIOT (LOcalization-Ready Internet Of Things), a novel system for mapping networked IoT devices to physical objects.
• Enable automatic generation of device control panels for both AR and non-AR clients.
• Evaluate the effectiveness of AR for IoT device control compared to traditional methods.

Main Methods:

• Utilize a semantic scene graph and a large language model for device-to-object mapping.
• Employ a declarative UI specification for automatic control panel generation.
• Conduct user studies to compare AR control with conventional smartphone app and remote control interfaces.

Main Results:

• Achieved a macro-averaged F1 score of 0.80 for digital-to-physical associations using only network metadata.
• Improved mapping performance to 0.88 F1 score by enriching metadata with physical attributes (location, materials, color, size).
• User study demonstrated significantly faster task completion, lower mental demand, and higher usability with AR control.

Conclusions:

• LORIOT system effectively maps networked IoT devices to physical objects using semantic scene graphs and LLMs.
• Spatially registered AR control significantly enhances user experience and task efficiency for IoT devices.
• AR-based control presents a viable and superior alternative to traditional methods for managing connected devices.