Neural network-based nonlinear equalizer using sight focal optimization for PCS 128 QAM 1 Tbit/s optical...

Area of Science:

• Optical communications
• Machine learning
• Signal processing

Background:

• High-speed coherent optical systems face challenges with nonlinear impairments.
• Conventional machine learning methods for nonlinear equalizers (NNLEs) can suffer from training inefficiencies.
• Overfitting and vanishing gradients are common issues with existing loss functions.

Purpose of the Study:

• To propose a novel sight focal loss (SFL) and SFL optimization-based neural network nonlinear equalizer (SFL-NNLE).
• To improve the training efficiency of NNLEs in high-speed optical systems.
• To mitigate overfitting and vanishing gradient problems inherent in conventional loss functions.

Main Methods:

• Developed a novel sight focal loss (SFL) optimization strategy.
• Implemented an SFL-NNLE, extending conventional machine learning approaches.
• Leveraged spatial information to focus on boundary-proximal hard samples for dynamic prioritization of misclassifications.

Main Results:

• Experimentally demonstrated the effectiveness of SFL-NNLE against nonlinear impairments.
• Achieved a 1.45 dB OSNR gain at the 24% SD-FEC threshold in a 118-GBaud DP-PCS 128 QAM BTB transmission.
• Realized a net bit rate exceeding 1 Tbps in back-to-back and 150 km SSMF transmissions.

Conclusions:

• The proposed SFL-NNLE effectively mitigates nonlinear impairments in high-speed optical systems.
• SFL optimization enhances training efficiency and addresses limitations of conventional loss functions.
• The SFL-NNLE enables ultra-high net bit rates exceeding 1 Tbps.