Two-Stream Modality-Based Deep Learning Approach for Enhanced Two-Person Human Interaction Recognition in Videos

Area of Science:

• Computer Vision
• Pattern Recognition
• Deep Learning

Background:

• Human Interaction Recognition (HIR) is crucial for applications like healthcare and surveillance.
• Existing video-based HIR systems struggle with complex actions, varied viewpoints, and environmental factors.
• Challenges include achieving satisfactory performance due to motion variations and data complexity.

Purpose of the Study:

• To propose a robust two-stream deep learning-based HIR system.
• To enhance the accuracy and reliability of human interaction recognition in videos.
• To address the limitations of current HIR methodologies.

Main Methods:

• A two-stream deep learning architecture utilizing skeleton and RGB data.
• Stream 1: YOLOv8-Pose for pose extraction, LSM modules, and dense layers.
• Stream 2: Segment Anything Model (SAM) for mesh generation, LSTM/GRU for long-range dependencies, and custom filtering.
• Feature fusion through concatenation followed by a classification module.

Main Results:

• Achieved 96.56% and 96.16% accuracy on two benchmark datasets.
• Demonstrated superior performance compared to existing human interaction recognition models.
• The custom filter function improved computational efficiency by removing irrelevant data.

Conclusions:

• The proposed two-stream deep learning system significantly advances human interaction recognition.
• The model's high accuracy validates its effectiveness and superiority.
• This approach offers a reliable solution for complex video-based interaction analysis.