Magnetic encoder error compensation based on a neural network-Kalman filtering hybrid framework

In industrial settings, magnetic encoders suffer from structural errors and random noise, making conventional angle demodulation methods that assume ideal sine-cosine signals hard to be both accurate and robust under complex conditions. Moreover, standard filters often fail in these scenarios because their static observation models and fixed noise parameters cannot adapt to structural distortions. To address this, this paper proposes a hybrid error compensation framework that integrates a neural network with an extended Kalman filter (NN-EKF). The neural network reconstructs signals that satisfy the quadrature relationship from noisy sine-cosine sequences sampled by the ADC and predicts the uncertainty parameters required by Kalman filtering. This provides the EKF with observation inputs and noise priors that are more consistent with its physical assumptions. Meanwhile, within an uncertainty-weighted multi-task learning framework, signal reconstruction and phase estimation are optimized jointly. Simulation results under mixed error conditions show that NN-EKF reduces the mean absolute error (MAE) by about 98% compared with arctangent demodulation and achieves lower MAE, RMSE, and maximum error than EKF, phase-locked loop, Kalman-Gradient Descent, and KalmanNet. On a hardware test platform, the proposed method achieves a maximum angle error of 0.0349°. Without target-sensor fine-tuning, NN-EKF reduces the RMSE by 84.2% and 83.1% on unseen tunneling magnetoresistance and Hall sensors. Furthermore, a lightweight lookup-table scheme constructed based on the NN-EKF output satisfies the real-time deployment requirements of embedded systems with only an additional error of ∼0.0022°, validating the engineering feasibility and practical potential of the proposed hybrid framework.