Deep Learning-Based Human Joint Localization Using mmWave Radar and Sequential Frame Fusion

Unlike conventional techniques relying typically on RGB and depth cameras, millimeter wave (mmWave) radars, especially Frequency-Modulated Continuous Wave (FMCW) radar, have significant advantages in human monitoring. These advantages are more pronounced when detecting human joint positions, as FMCW radars rely solely on collected point cloud data. Compared with camera-based systems, this technology is efficient because it can operate in low-light conditions, and does not present a privacy threat. However, conventional approaches employing FMCW radar, such as voxel-based gathering techniques, still have limitations in optimally capturing spatiotemporal features, making it difficult to detect human joint points with high accuracy. To overcome this obstacle, we propose a novel framework that introduces sequential concatenation of multiple frames in the preprocessing stage to maintain spatiotemporal continuity, thereby improving feature representation. Furthermore, we design a deep learning architecture consisting of Convolutional Neural Network (CNN) for spatial feature extraction, Transformer for capturing long-term dependencies, and Bidirectional Long Short-Term Memory (Bi-LSTM) for temporal sequence modeling. The proposed approach makes optimal use of spatiotemporal data, improves the accuracy and reliability in detecting human joint positions, and accelerates model convergence. Through experimental validation, this method shows a Mean Absolute Error (MAE) of 1.77 cm and a Root Mean Squared Error (RMSE) of 2.92 cm between the actual human joint positions and the predicted ones. This reaffirms the effectiveness of the model in estimating human joint positions.