Understanding action concepts from videos and brain activity through subjects' consensus

Area of Science:

• Neuroscience
• Computer Vision
• Cognitive Science

Background:

• Action recognition in computer vision often relies on explicit visual cues.
• Implicit action concepts, understood by humans but not always visually explicit, pose a challenge for AI.
• Understanding brain activity during visual perception can offer new insights into action recognition.

Purpose of the Study:

• To investigate brain activity (EEG) associated with complex visual tasks.
• To enhance computer vision's ability to recognize actions from videos, including implicitly represented ones.
• To explore the potential of using human brain responses to improve AI-driven action recognition.

Main Methods:

• Collected multi-modal electroencephalography (EEG) and video data from a large sample of subjects.
• Utilized the Moments in Time dataset, featuring videos with implicit action references.
• Developed a computational pipeline to transfer knowledge from EEG data to video analysis.

Main Results:

• Discovered a consistent agreement in brain activity (subjects consensus) among individuals viewing the same video stimuli.
• Demonstrated that incorporating EEG-derived insights significantly boosts action recognition performance.
• Showcased the effectiveness of this approach on complex, implicitly defined action concepts.

Conclusions:

• Human brain activity patterns contain valuable information for improving computer vision.
• Subject consensus in EEG provides a robust signal for understanding visual action concepts.
• This neuro-computational approach offers a promising direction for advancing AI-based video understanding.