Leveraging Hand-Object Interactions in Assistive Egocentric Vision.

Area of Science:

• Computer Vision
• Human-Computer Interaction
• Assistive Technology

Background:

• Egocentric vision systems aim to enhance visual information access for blind individuals.
• Current systems struggle with identifying user interest and object presence due to camera aiming challenges and unreliable gaze data.
• Blind users often position their hands near objects of interest for interaction or camera aiming.

Purpose of the Study:

• To develop a method that utilizes hand presence as contextual information for object recognition in egocentric vision.
• To improve the accuracy of object localization and classification for assistive technologies.
• To address the limitations of gaze-based attention inference in egocentric vision.

Main Methods:

• A novel object recognition method was proposed, integrating hand segmentation outputs into a convolutional neural network.
• The network architecture included separate output layers for object localization and classification.
• The approach was evaluated using egocentric datasets collected from both sighted and blind individuals.

Main Results:

• Hand-priming significantly improved object localization accuracy compared to other hand-encoding methods.
• The proposed method achieved classification performance comparable to state-of-the-art techniques using bounding boxes, despite using only object centers and labels.
• Demonstrated effectiveness across datasets from diverse user groups.

Conclusions:

• Leveraging hand cues is a promising strategy for enhancing egocentric object recognition, particularly for assistive applications.
• The proposed hand-priming method offers a robust and accurate solution for object localization and classification.
• This approach has the potential to significantly improve the quality of life for blind individuals by increasing access to visual information.