Skeleton Driven Action Recognition Using an Image-Based Spatial-Temporal Representation and Convolution Neural...

Area of Science:

• Robotics and Human-Computer Interaction
• Developmental Psychology
• Computer Vision

Background:

• Children with Autism Spectrum Disorder (ASD) often struggle with social interaction.
• Technological tools, including social robots, are being developed to support children with ASD.
• Current social robots lack the ability to adapt their behavior due to a lack of user action recognition.

Purpose of the Study:

• To propose a method for real-time detection of typical and stereotypical actions in children with ASD.
• To investigate the impact of joint sequence representation on the performance of action recognition models.
• To develop an accurate classification system for behavioral patterns in children with ASD.

Main Methods:

• Utilized Intel RealSense and Nuitrack SDK to extract user joint coordinates.
• Mapped temporal and spatial joint dynamics onto a color image-based representation.
• Conducted experiments varying joint order in sequences and analyzed using statistical methods.
• Trained a Convolutional Neural Network (CNN) for behavior classification.

Main Results:

• Found statistically significant differences in model performance based on the sequence of joints.
• The developed CNN model achieved a mean accuracy of 92.4% in classifying typical and stereotypical actions.
• The entire detection and classification pipeline operated at an average of 31 FPS.

Conclusions:

• The order of joint representation is a critical factor influencing the performance of action recognition models for children with ASD.
• The proposed method offers an effective approach for real-time behavioral analysis in children with ASD.
• This technology can enhance the adaptive capabilities of social robots and support systems for individuals with ASD.