Nonlinear compensation using a robust neural network for noisy samples in a high-speed coherent optical transceiver

Area of Science:

• Optical communications
• Signal processing
• Machine learning

Background:

• Nonlinear impairments limit transmission capacity in coherent optical transceivers.
• Current data-driven nonlinear equalizers (NLEs) face challenges with training data quality and framework compatibility.
• Neural network-based nonlinear equalizers (NNLEs) offer potential but require robust training methodologies.

Purpose of the Study:

• To address the deficiencies of existing NLEs and improve their practical applicability.
• To develop a robust NNLE architecture capable of handling noisy training datasets.
• To enhance the performance and reliability of nonlinear impairment compensation in high-speed optical communication systems.

Main Methods:

• Proposed a noisy-samples-robust NN (NSNN) based NLE architecture.
• Extended meta-training methods to improve NNLE training.
• Employed bilevel optimization with meta-learning for noisy sample purification.

Main Results:

• The NSNN achieved an OSNR gain of over 1.5 dB at a 1 Tbps transmission rate for 128 QAM.
• Demonstrated effective performance in high-speed coherent optical transceivers with nonlinear impairments.
• Showcased stable performance gains across various conditions with acceptable complexity.

Conclusions:

• The proposed NSNN provides a reliable solution for nonlinear impairment compensation, especially with noisy training data.
• Achieved superior and more stable generalizability compared to common NLEs.
• Offers a practical advancement for enhancing transmission capacity in optical networks.