LCwmcaR: Learning Cross-Window Cross-Modality Correlation-Aware Representation for Human Activity Recognition.

Area of Science:

• Computer Science
• Artificial Intelligence
• Signal Processing

Background:

• Deep learning (DL) shows promise for human activity recognition (HAR).
• Current HAR methods struggle with spatial distribution and spatial-temporal (ST) dependencies.
• Existing models lack cross-window interaction, leading to feature inconsistencies.

Purpose of the Study:

• To propose a novel framework, LCwmcaR, for enhanced human activity recognition.
• To address limitations in modeling ST dependencies and cross-window feature learning.
• To improve the robustness and accuracy of HAR systems.

Main Methods:

• Developed a dual-branch network (LCwmcaR) using Mamba and CNN to model temporal and spatial information.
• Introduced a learnable temporal 2-dimensionalization (LT2D) strategy for integrating local and global ST dependencies.
• Implemented a cross-window correlation-aware feature representation generation (CrwcaFRGen) module for robust feature extraction.

Main Results:

• LCwmcaR effectively models spatial distribution and ST dependencies.
• The LT2D strategy creates integrated 2-D representations of temporal patterns.
• The CrwcaFRGen module generates robust features by correlating multiple window representations.
• Experimental results show LCwmcaR significantly outperforms state-of-the-art methods on four public datasets.

Conclusions:

• LCwmcaR offers a significant advancement in deep learning-based HAR.
• The proposed framework effectively tackles key challenges in HAR, including ST dependencies and feature consistency.
• LCwmcaR demonstrates superior performance, paving the way for more accurate and reliable human activity recognition systems.