Clustered Federated Spatio-Temporal Graph Attention Networks for Skeleton-Based Action Recognition

Area of Science:

• Computer Science
• Artificial Intelligence
• Machine Learning

Background:

• Federated learning (FL) for skeleton-based action recognition is challenging due to client heterogeneity, leading to model drift and convergence issues with standard methods like FedAvg.
• Existing FL approaches struggle to maintain stable performance when clients exhibit significant variations in their data distributions and model parameters.

Purpose of the Study:

• To introduce a novel clustered FL framework, Clustered Federated Spatio-Temporal Graph Attention Networks (CF-STGAT), designed to address client heterogeneity in skeleton-based action recognition.
• To enhance the stability and accuracy of FL models by dynamically grouping clients and performing attention-weighted inter-cluster fusion.

Main Methods:

• Leveraged attention-derived spatio-temporal statistics from local Spatio-Temporal Graph Attention Network (STGAT) models to dynamically group clients.
• Implemented a server-side process involving extraction, normalization, and PCA projection of multi-head parameter-based attention descriptors for K-means clustering.
• Utilized attention-similarity weighting to compute a global reference for regularizing cluster models via a lightweight fusion step, keeping local training unchanged.

Main Results:

• CF-STGAT consistently outperformed strong FL baselines on the NTU RGB+D 60/120 datasets, achieving significant absolute top-1 accuracy gains over FedAvg.
• Demonstrated improved performance across different evaluation settings (X-Sub/X-Setup), with notable gains of +0.84/+4.09 on NTU 60 and +7.98/+4.18 on NTU 120.
• Observed smoother per-client training trajectories and lower terminal test loss, indicating enhanced model stability and convergence.

Conclusions:

• Attention-guided clustering and inter-cluster fusion are effective complementary strategies for mitigating within-group variance and cross-cluster divergence in FL.
• The CF-STGAT framework offers a robust solution for skeleton-based action recognition under strong client heterogeneity without altering local training procedures.
• The proposed method effectively enhances federated learning performance by leveraging attention mechanisms for dynamic client grouping and model alignment.