Secure Cooperative Communications in 6G Networks: A Constrained Hierarchical Reinforcement Learning Framework with...

With the rapid evolution toward 6G networks, ensuring robust physical layer security (PLS) in highly dynamic and heterogeneous wireless environments has become a key challenge. Traditional security methods often struggle to adapt to time-varying channels, especially in the absence of perfect channel state information. Furthermore, the dynamic nature of node selection and power allocation in heterogeneous networks creates a complex hybrid action space operating across multiple timescales, significantly complicating the design of efficient and adaptive security strategies. To address this, this paper proposes a novel constrained hierarchical reinforcement learning (CHRL) framework for secure cooperative communications in next-generation wireless systems. The framework is designed to optimize secrecy performance within a hybrid action space comprising both discrete node selection and continuous power allocation, operating at different timescales. By hierarchically decoupling the joint optimization problem, the upper layer performs risk-aware node selection to maximize long-term secrecy capacity (SC) while guaranteeing a stable and secure link. At the lower layer, we develop a constrained MiniMax Multi-objective Deep Deterministic Policy Gradient (M3DDPG) algorithm that optimizes power allocation considering worst-case conditions. Lagrange multipliers are integrated to enforce a strictly positive SC constraint throughout transmission, effectively preventing security outages. Simulation results under time-varying Rayleigh fading channels demonstrate that the proposed CHRL framework outperforms existing HRL methods, achieving up to 17% improvement in SC while strictly maintaining security constraints. These results validate the effectiveness of the proposed approach for enhancing PLS in next-generation cooperative wireless networks.