Efficient neural network equalization via automatic grouping-enabled non-uniform quantization for 120-Gb/s PAM-8...

Area of Science:

• Optical Communications
• Machine Learning
• Signal Processing

Background:

• High-speed optical communication systems rely on efficient equalization techniques.
• Directly modulated laser (DML)-based PAM-8 systems present equalization challenges.
• Quantization of neural networks impacts hardware efficiency and performance.

Purpose of the Study:

• To propose and evaluate an automatic grouping-enabled non-uniform quantization (AGNQ) approach for neural network equalization.
• To compare AGNQ with traditional uniform and power-of-two quantization methods.
• To assess the impact of AGNQ on performance and memory usage in a 120 Gb/s optical system.

Main Methods:

• Developed an automatic grouping-enabled non-uniform quantization (AGNQ) feedforward neural network (FNN).
• Implemented and tested AGNQ-FNN in a 120 Gb/s, 10 km DML-based PAM-8 IM/DD system.
• Compared AGNQ-FNN bit error rate (BER) and memory usage against uniform quantized FNN (UQ-FNN) and additive power-of-two quantized FNN (APoT-FNN).

Main Results:

• AGNQ-FNN achieved comparable BER performance below the 7% HD-FEC threshold using only 4-bit quantization.
• UQ-FNN required 10-bit and APoT-FNN required 8-bit quantization for similar performance.
• AGNQ-FNN reduced weight and bias memory usage by 47.9% and 35.5% compared to UQ-FNN and APoT-FNN, respectively.

Conclusions:

• AGNQ-FNN offers a hardware-friendly solution for efficient equalization in high-speed optical communication.
• The proposed quantization method significantly reduces memory footprint without compromising performance.
• AGNQ demonstrates strong potential for practical implementation in short-reach optical systems.