Cross-Modal Feature Adapter for Few-Shot Human Activity Recognition

Recent years have witnessed outstanding success of deep learning in sensor-based human activity recognition (HAR), spanning a wide range of real-world applications like healthcare management, fitness tracking, and fall detection. However, sensor data annotation scarcity still remains a main challenge unresolved, which requires human annotators to take a long-term and tedious observation to segment and timestamp sensor samples meticulously, hampering the wide use of deep learning models, especially in few-shot HAR scenario. To handle such issue, this paper introduces a cross-modal data augmentation, by exploiting activity label text as key words to search for activity-related images to construct an augmented dataset. On this basis, a new cross-modal feature adapter is designed, which fine-tunes a pre-trained CLIP image encoder to effectively align image-sensor pairs. Through a learnable residual ratio, it may adaptively blend the old knowledge inherited from the original zero-shot CLIP with the new knowledge adapted from few-shot training samples, which makes training converges faster whilst forming a streamlined time series sensor encoder. Extensive experiments and ablation studies are performed on three public HAR benchmarks. The experimental results demonstrate that the proposed method outperforms existing state-of-the-art HAR baselines under all few-shot scenarios. A practical on-device inference latency is provided.