Attention-aided partial bidirectional RNN-based nonlinear equalizer in coherent optical systems.

Area of Science:

• Optical communication systems engineering
• Signal processing
• Machine learning applications in telecommunications

Background:

• Bidirectional recurrent neural networks (BRNNs) are used for nonlinear equalization in optical systems.
• Attention mechanisms can improve the interpretability and efficiency of neural network models.
• High-baud-rate systems like DP-64QAM face significant nonlinear impairments.

Purpose of the Study:

• To develop a novel, complexity-reduced nonlinear equalizer for coherent optical communication systems.
• To investigate the contribution of input symbols and hidden representations using attention mechanisms.
• To evaluate the performance of an attention-aided partial BRNN equalizer using LSTM and GRU units.

Main Methods:

• Implementation of a partial BRNN-based nonlinear equalizer incorporating an attention mechanism.
• Evaluation using Long Short-Term Memory (LSTM) and Gated Recurrent Unit (GRU) cells.
• Testing in a single-channel DP-64QAM 30Gbaud system over 20 × 50 km standard single-mode fiber (SSMF).
• Comparison of computational complexity (real multiplications per symbol - RMpS) and Q-factor performance against a baseline equalizer.

Main Results:

• The attention-aided equalizer achieved a complexity reduction of approximately 56.16% in RMpS.
• Maintained comparable Q-factor performance to the baseline equalizer.
• Outperformed the baseline by 0.2dB to 0.25dB at optimal transmit power and 0.3dB to 0.45dB in more nonlinear regions for similar complexity.

Conclusions:

• The proposed attention-aided partial BRNN equalizer offers a significant reduction in computational complexity.
• It achieves competitive or superior performance compared to traditional equalizers, especially under nonlinear conditions.
• This approach provides an efficient solution for nonlinear equalization in advanced optical communication systems.