Efficient neural network equalization via automatic grouping-enabled non-uniform quantization for 120-Gb/s PAM-8...

An automatic grouping-enabled non-uniform quantization (AGNQ) approach is proposed for efficient neural network (NN) equalization in a 120 Gb/s 10 km directly modulated laser (DML)-based PAM-8 intensity-modulation/direct-detection (IM/DD) system. Compared with traditional uniform quantized feedforward neural network (UQ-FNN) and the additive power-of-two quantized FNN (APoT-FNN) approaches, AGNQ-FNN achieves comparable bit error rate (BER) performance below the 7% hard-decision forward error correction (HD-FEC) threshold using only 4-bit weight and bias quantization, whereas UQ-FNN and APoT-FNN require 10-bit and 8-bit quantization, respectively. Furthermore, when approaching the BER performance of an un-quantized FNN baseline, AGNQ-FNN reduces weight and bias memory usage by 47.9% and 35.5% compared with UQ-FNN and APoT-FNN, respectively. These results highlight the potential of AGNQ-FNN as a hardware-friendly equalizer for high-speed short-reach optical communication systems.