The AM4I Architecture and Framework for Multimodal Interaction and Its Application to Smart Environments

Technologies, such as smart sensors, actuators, and other kinds of devices, are often installed in our environments (e.g., our Homes) and available to integrate our daily lives. Despite their installation being motivated by the pursuit of automation and increased efficiency, making these environments usable, acceptable and enjoyable in a sustainable, energy efficient way is not only a matter of automation. Tackling these goals is a complex task demanding the combination of different perspectives including building and urban Architecture, Ubiquitous Computing and Human-Computer Interaction (HCI) to provide occupants with the means to shape these environments to their needs. Interaction is of paramount relevance in the creation of adequate relations of users with their environments, but it cannot be seen independently from the ubiquitous sensing and computing or the environment's architecture. In this regard, there are several challenges to HCI, particularly in how to integrate this multidisciplinary effort. Although there are several solutions to address some of these challenges, the complexity and dynamic nature of the smart environments and the diversity of technologies involved still present many challenges, particularly for its development. In general, the development is complex, and it is hard to create a dynamic environment providing versatile and adaptive forms of interaction. To participate in the multidisciplinary effort, the development of interaction must be supported by tools capable of facilitating co-design by multidisciplinary teams. In this article, we address the development of interaction for complex smart environments and propose the AM4I architecture and framework, a novel modular approach to design and develop adaptive multiplatform multilingual multi-device multimodal interactive systems. The potential of the framework is demonstrated by proof-of-concept applications in two different smart environment contexts, non-residential buildings and smart homes.