Cross-Modal Federated Human Activity Recognition.

Area of Science:

• Computer Science
• Artificial Intelligence
• Machine Learning

Background:

• Federated human activity recognition (FHAR) enables privacy-preserving collaborative learning of activity models.
• Existing FHAR methods often assume unimodal data, limiting application in real-world scenarios with diverse local data modalities.
• Cross-modal federated human activity recognition (CM-FHAR) addresses scenarios where clients possess data from different sources (e.g., motion vs. visual).

Purpose of the Study:

• To introduce and address the challenges of cross-modal federated human activity recognition (CM-FHAR).
• To develop a novel network (MCARN) capable of learning both global and private activity classifiers across heterogeneous data modalities.
• To overcome modality imbalance issues in federated learning settings.

Main Methods:

• Propose the Modality-Collaborative Activity Recognition Network (MCARN) with modality-agnostic and modality-specific feature learning using altruistic and egocentric encoders.
• Employ a separation loss and adversarial modality discriminator in hyper-sphere for feature learning.
• Introduce an angular margin adjustment scheme for modality imbalance and a relation-aware global-local calibration mechanism.

Main Results:

• MCARN effectively learns a shared global classifier and modality-dependent private classifiers.
• The proposed methods successfully address distributive common cross-modal feature learning, modality-dependent discriminate feature learning, and modality imbalance.
• Achieved state-of-the-art performance on both modality-balanced and modality-imbalanced datasets.

Conclusions:

• MCARN provides a robust solution for cross-modal federated human activity recognition.
• The approach enhances the practicality and scalability of federated learning for human activity recognition.
• This work paves the way for more widespread deployment of HAR models on diverse edge devices.