Neural network architectures for optical channel nonlinear compensation in digital subcarrier multiplexing systems

In this work, we propose to use various artificial neural network (ANN) structures for modeling and compensation of intra- and inter-subcarrier fiber nonlinear interference in digital subcarrier multiplexing (DSCM) optical transmission systems. We perform nonlinear channel equalization by employing different ANN cores including convolutional neural networks (CNN) and long short-term memory (LSTM) layers. First, we develop a fiber nonlinearity compensation for DSCM systems based on a fully-connected network across all subcarriers. In subsequent steps, and borrowing from the perturbation analysis of fiber nonlinearity, we gradually upgrade proposed designs towards modular structures with better performance-complexity advantages. Our study shows that putting proper macro structures in design of ANN nonlinear equalizers in DSCM systems can be crucial in development of practical solutions for future generations of coherent optical transceivers.