IoT-LLM: A framework for enhancing large language model reasoning from real-world sensor data

Large language models (LLMs) excel in textual tasks but often struggle with physical-world reasoning tasks. Inspired by human cognition-where perception is fundamental to reasoning-we explore augmenting LLMs with enhanced perception abilities using Internet of Things (IoT) data and pertinent knowledge. In this work, we systematically study LLMs' capability to address IoT-sensory tasks, by augmenting their perception and knowledge base, and then propose a unified framework, IoT-LLM, to enhance such capability. In IoT-LLM, we customize three steps: preprocessing IoT data into suitable formats, expanding LLMs' knowledge via IoT-oriented retrieval-augmented generation, and activating LLMs' commonsense knowledge through chain-of-thought prompting. We design a benchmark comprising five real-world tasks with varying data types and reasoning complexities to evaluate the performance of IoT-LLM. Experimental results reveal that IoT-LLM significantly improves the performance of IoT-sensory task reasoning of LLMs, with models such as GPT-4o-mini showing a 49.4% average improvement over previous methods.