IoT-LLM: A framework for enhancing large language model reasoning from real-world sensor data.

Area of Science:

• Artificial Intelligence
• Internet of Things

Background:

• Large language models (LLMs) demonstrate strong text capabilities but lack physical-world reasoning.
• Human cognition relies on perception for effective reasoning.

Purpose of the Study:

• To augment LLMs with enhanced perception using Internet of Things (IoT) data and knowledge.
• To systematically evaluate LLMs' performance on IoT-sensory tasks.
• To propose a unified framework, IoT-LLM, for improving IoT-sensory reasoning.

Main Methods:

• Developed the IoT-LLM framework with three key steps: data preprocessing, knowledge expansion via retrieval-augmented generation, and commonsense activation using chain-of-thought prompting.
• Created a benchmark of five real-world IoT-sensory tasks with diverse data types and reasoning complexities.
• Evaluated LLM performance using the benchmark.

Main Results:

• The IoT-LLM framework significantly enhances LLMs' performance in IoT-sensory task reasoning.
• Models like GPT-4o-mini achieved a 49.4% average improvement over prior methods.
• The framework demonstrated effectiveness across tasks with varying complexities.

Conclusions:

• Augmenting LLMs with IoT data and knowledge through the IoT-LLM framework is effective for physical-world reasoning.
• The proposed framework offers a viable solution for bridging the gap between LLM textual abilities and real-world sensory understanding.
• Future work can explore further refinements of the framework for broader applications.