Hybrid pruning for nonlinear equalization in short-reach high-speed optical links

Nonlinear impairments from transceiver components in short-reach optical communication systems, such as drivers, TIAs, and external modulators, pose significant challenges for achieving high data rates with low latency. Volterra Nonlinear Equalization (VNLE) can mitigate these impairments but suffers from higher computational complexity, rendering it impractical for energy-efficient and low-cost data center applications. We propose a Hybrid Pruned VNLE (HP-VNLE) that combines structured and unstructured kernel reduction to significantly reduce complexity while maintaining low bit error rate (BER) performance. This study reveals that most non-significant taps in the third-order unpruned VNLE consist of at least three different index terms, justifying the necessity of pruning to remove taps with minimal contribution to nonlinearity mitigation. Experimental results demonstrate that the HP-VNLE's performance highly depends on the initial tap configuration derived from the proposed structured kernel reduction so-called center-spread (CS)VNLE. Moreover, combining the third-order HP-VNLE with a linear equalizer and a 1-tap decision feedback equalizer (DFE) yields superior BER results compared to unpruned VNLE. This hybrid approach reduces the computational complexity and enhances system performance, offering a range of promising solutions for future optical communication systems where energy efficiency, low complexity, and high data rates must be optimized for various application scenarios.