Weight-adaptive joint mixed-precision quantization and pruning for neural network-based equalization in short-reach...

Area of Science:

• Optical Communications
• Machine Learning
• Signal Processing

Background:

• Neural network (NN)-based equalizers excel at mitigating nonlinear impairments in intensity-modulated direct detection (IM/DD) systems.
• High computational complexity (CC) of NNs hinders their real-time application in optical receiver design.

Purpose of the Study:

• To propose a novel weight-adaptive, mixed-precision quantization and pruning approach to reduce the CC of NN-based equalizers.
• To enable real-time processing using only integer arithmetic, thereby reducing hardware resource consumption.

Main Methods:

• A weight-adaptive joint mixed-precision quantization and pruning strategy is introduced.
• NN connections are pruned or quantized to a specific bit-width, creating a hybrid compressed sparse network.
• The method exclusively utilizes integer arithmetic, avoiding floating-point operations.

Main Results:

• The approach was validated on a 50-Gb/s, 25-km PAM-4 IM/DD link employing a directly modulated laser (DML).
• Approximately 80% memory savings were achieved at minimum network size without performance degradation compared to standard NNs.
• Quantization proved more effective for over-parameterized NNs than for NNs at minimum size.

Conclusions:

• The proposed method effectively reduces the computational complexity and hardware resource requirements of NN-based equalizers.
• This technique facilitates the real-time implementation of advanced NN equalizers in optical communication systems.
• Mixed-precision quantization and pruning offer a viable solution for optimizing NN performance in resource-constrained environments.