Interaction Detection with Depth Sensing and Body Tracking Cameras in Physical Rehabilitation

This article is part of the Focus Theme of Methods of Information in Medicine on "Methodologies, Models and Algorithms for Patients Rehabilitation".

This paper presents a camera based method for identifying the patient and detecting interactions between the patient and the therapist during therapy. Detecting interactions helps to discriminate between active and passive motion of the patient as well as to estimate the accuracy of the skeletal data.

Continuous face recognition is used to detect, recognize and track the patient with other people in the scene (e.g. the therapist, or a clinician). We use a method based on local binary patterns (LBP). After identifying users in the scene we identify interactions between the patient and other people. We use a depth map/point cloud for estimating the distance between two people. Our method uses the association of depth regions to user identities and computes the minimal distance between the regions.

Our results show state-of-the-art performance of real-time face recognition using low-resolution images that is sufficient to use in adaptive systems. Our proposed approach for detecting interactions shows 91.9% overall recognition accuracy what is sufficient for applications in the context of serious games. We also discuss limitations of the proposed method as well as general limitations of using depth cameras for serious games.

We introduced a new method for frame-by-frame automated identification of the patient and labeling reliable sequences of the patient's data recorded during rehabilitation (games). Our method improves automated rehabilitation systems by detecting the identity of the patient as well as of the therapist and by detecting the distance between both over time.