Nonlinear compensation for ultra-high symbol rate PDM-WDM systems based on second-order perturbation theory

In high-speed, large-capacity, and long-haul optical transmission scenarios, the elevating symbol rate, increasing number of channels, and extending transmission distance significantly enhance Kerr nonlinearity, which severely degrading system performance. While first-order perturbation-based nonlinear compensation has been widely studied and combined with machine learning techniques, it faces performance limitations due to the growing prominence of intra- and inter-channel higher-order nonlinear interference. In this work, for what we believe to be the first time, we propose a second-order perturbation theory model for the polarization division multiplexing (PDM) wavelength division multiplexing (WDM) system. We begin by characterizing the joint effect of first- and second-order nonlinear impairments within and between WDM channels using triplets and quintuplets, while accounting for nonlinear crosstalk between preceding and subsequent symbols in the target channel-especially in the scenario of ultra-high symbol rate. Subsequently, the characterized feature is fed into a convolutional bidirectional long short-term memory (CNN-BiLSTM) neural network assisted with knowledge distillation and transfer learning, achieving a significant reduction in the implementation complexity while requiring just a fraction (down to 10%) of the initial training data to accomplish the compensation of nonlinear impairment in different launch power scenarios. Simulation results show that, in an 11×128 GBaud PDM-WDM system, the proposed scheme improves the Q-factor by 0.64 dB and 0.36 dB compared with linear and digital back propagation with 40 steps per span (DBP-40StPS) compensation schemes, respectively, for 16QAM over 1600km, and by 0.57 dB and 0.32dB for 64QAM over 400km. In a 3×64 GBaud 16QAM experimental system over 1400km, the proposed scheme provides Q-factor gains of 0.71 dB and 0.36 dB compared with the linear and DBP-40Stps schemes, respectively. Furthermore, it reduces the computational complexity by 62.7% and 55.7% in the 16QAM and 64QAM simulation systems, and by 68.6% in the 16QAM experimental system, compared with the DBP-40StPS scheme.