Performance vs. complexity in NN pre-distortion for a nonlinear channel

Optical communications at high bandwidth and high spectral efficiency rely on the use of a digital-to-analog converter (DAC). We propose the use of a neural network (NN) for digital pre-distortion (DPD) to mitigate the quantization and bandlimited impairments from a DAC in such systems. We experimentally validate our approach with a 64 Gbaud 8-level pulse amplitude modulation (PAM-8) signal. We examine the NN-DPD training with both direct and indirect learning methods. We compare the performance with typical Volterra, look-up table (LUT) and linear DPD solutions. We sweep regimes where nonlinear quantization becomes more prominent to highlight the advantages of NN-DPD. The proposed NN-DPD trained via direct learning outperforms the Volterra, LUT and linear DPDs by almost 0.9 dB, 1.9 dB and 2.9 dB, respectively. We find that an indirect learning recurrent NN offers better performance at the same complexity as Volterra, while a direct learning recursive NN pushes performance to a higher level than a Volterra can achieve.