Low-complexity MLSE for coherent optical fiber transmission systems with symbol correlation

This study demonstrates the effectiveness of a simplified maximum likelihood sequence estimation (MLSE) approach in enhancing the performance of coherent optical fiber transmission systems, particularly under conditions of linear and nonlinear symbol correlation. To counter the high complexity of traditional MLSE in high-speed systems, we introduce what we believe to be a novel simplification method. This approach replaces the conventional channel impulse response (CIR) convolution with a look-up table (LUT) and simplifies the Euclidean distance (ED) calculations. Additionally, we provide a detailed analysis of algorithmic operations, complementary metal oxide semiconductor (CMOS) transistor count, and power consumption across various technology nodes. The proposed simplified MLSE scheme has the potential to reduce implementation complexity to approximately 5% of the conventional approach. The proposed low-complexity approach was verified with both simulations and experiments. The results show that the performance loss is less than 0.3 dB in Q-factors for both linear and nonlinear bandwidth-constrained transmission scenarios.