Holistic-Guided Disentangled Learning With Cross-Video Semantics Mining for Concurrent First-Person and Third-Person...

Area of Science:

• Computer Vision
• Artificial Intelligence
• Human-Computer Interaction

Background:

• Wearable devices drive demand for first-person activity recognition.
• Existing datasets often overlook third-person activities occurring alongside first-person actions.
• Recognizing concurrent activities enhances situational awareness for camera wearers.

Purpose of the Study:

• Introduce a novel dataset, PolyU concurrent first- and third-person (CFT) Daily, for concurrent first- and third-person activity recognition (CFT-AR).
• Develop a robust method to analyze and recognize both first- and third-person activities simultaneously from wearable camera data.
• Address challenges like temporal asynchronism and appearance gaps in concurrent activity recognition.

Main Methods:

• Utilized a 3-D convolutional neural network for holistic scene-level feature extraction.
• Employed attention-based modules and self-knowledge distillation (SKD) to mine shared and unique semantics.
• Leveraged holistic features to guide disentangled learning of instance-level (person-level) features.

Main Results:

• The proposed method achieves state-of-the-art performance on the PolyU CFT Daily dataset.
• Demonstrated effectiveness in capturing both holistic scene context and local person-specific cues.
• Successfully addressed temporal asynchronism and appearance gaps between concurrent activities.

Conclusions:

• The developed approach provides comprehensive and discriminative patterns for CFT-AR.
• The PolyU CFT Daily dataset presents unique challenges and facilitates advancements in the field.
• This research is crucial for improving understanding of complex, real-world scenarios captured by wearable devices.