An interaction-embedded HMM framework for human behavior understanding: with nursing environments as examples

This paper presents an interaction-embedded hidden Markov model (IE-HMM) framework for automatically detecting and classifying individual human behaviors and group interactions. The proposed framework comprises a switch control (SC) module, an individual duration HMM (IDHMM) module, and an interaction-coupled duration HMM (ICDHMM) module. By analyzing the relative distances between the various participants in each scene, and monitoring the duration for which these distances are maintained, the SC module assigns each participant to an individual behavior unit (comprising a single participant) or an interaction behavior unit (comprising two or more participants). The individual behavior units are passed to the IDHMM module, which classifies the corresponding human behavior in accordance with the pose, motion, and duration information using duration HMM (DHMM). Similarly, the interaction behavior units are dispatched to the ICDHMM module, where the corresponding interaction mode is classified using an integrated scheme comprising multiple coupled-duration HMM (CDHMM), in which each state has an embedded coupled HMM (CHMM). The validity of the IE-HMM framework is confirmed by analyzing the human actions and interactions observed in a nursing home environment. The results confirm that the atomic behavior unit concept embedded in the SC module enables the IE-HMM framework to recognize multiple concurrent actions and interactions within a single scene. Overall, it is shown that the proposed framework has a recognition performance of 100% when applied to the analysis of individual human actions and 95% when applied to that of group interactions.