Intelligent and adaptive intra-channel and inter-channel fiber nonlinearity compensation in coherent optical...

Fiber nonlinearity mitigation is a crucial technology for extending transmission reach and increasing channel capacity in high-baud rate wavelength division multiplexing (WDM) systems. In this work, we propose a novel, to the best of our knowledge, architecture that combines learned modified digital back-propagation (L-MDBP) to compensate for intra-channel nonlinearity and a two-stage decision-directed least mean square (DDLMS) adaptive equalizer to mitigate inter-channel nonlinearity. By leveraging globally optimized model parameters and adaptive channel estimation, the proposed scheme achieves superior performance and lower computation complexity compared with conventional DBP. Specifically, in an 8 × 64 Gbaud 16-ary quadrature amplitude modulation (16QAM) experimental system over 1600 km of standard single-mode fiber (SSMF), our approach shows a 0.30-dB Q²-factor improvement and a complexity reduction of 82.3% compared with DBP with 8 steps per span (SPS). Furthermore, we enhance the adaptability of the architecture by introducing an online transfer learning (TL) technique, which requires only 2% of initial training epochs.