Unified dual-decoder autoencoder for federated multi-modal clustering with arbitrary modality missingness

Federated multi-modal clustering (FedMMC) aims to collaboratively group data across multiple clients without sharing raw data. However, most existing methods still adopt restrictive client assumptions, considering clients as single-modal, fully multi-modal, or only a mixture of these two types, which is still unrealistic in practice. In real federated scenarios, clients often possess arbitrary subsets of modalities, leading to client-level arbitrary modality missingness. To address this issue, we propose a Unified Dual-Decoder Autoencoder framework for FedMMC. It uses a dual-decoder architecture to disentangle modality-shared and modality-specific information, and introduces modality-specific one-hot indicator vectors with an evaluator-guided fusion module to adaptively integrate heterogeneous modality subsets. In addition, we design a client-configuration-guided modality-missing simulation strategy with sequential parameter propagation to improve robustness under arbitrary modality-missing settings, and develop a mutual information-guided weighted aggregation strategy for global optimization. Extensive experiments on four datasets demonstrate that our method consistently outperforms existing methods in heterogeneous federated settings. The source code is available at https://github.com/xzy1095/UUDA.