Development and evaluation of a hand tracker using depth images captured from an overhead perspective

Area of Science:

• Computer Vision
• Rehabilitation Engineering
• Human-Computer Interaction

Background:

• The COACH system, an assistive technology for people with dementia, previously faced limitations in hand tracking during clinical trials.
• Accurate hand tracking is crucial for intelligent assistive technologies that support daily living activities.

Purpose of the Study:

• To develop and evaluate a robust hand tracker using single overhead depth images for the COACH system.
• To overcome the limitations of previous hand tracking methods used in the COACH system.

Main Methods:

• A random decision forest classifier was trained on approximately 5000 manually labeled images.
• Hand positions were translated into task actions based on proximity to environmental objects.
• Performance was evaluated using ~24,000 images from 41 participants and compared to a previous color-based tracker.

Main Results:

• The depth-based tracker achieved precision of 0.994 and recall of 0.938.
• This significantly outperformed the previous color-based tracker (precision 0.981, recall 0.822) on current data.
• Performance also surpassed the previous study's results for the color tracker (precision 0.989, recall 0.466).

Conclusions:

• The enhanced tracking performance validates the integration of the depth-based tracker into the COACH system for unsupervised trials.
• This advancement supports the development of intelligent assistive technologies for cognitive disabilities, promoting aging-in-place.
• Robust depth-based motion tracking has broader applications in assistive technologies, gaming, and automated assessments.