Towards automated joint detection in sleep studies: bridging clinical insight and artificial intelligence

To develop and evaluate artificial intelligence methods for detecting upper-body joint positions from video recordings of individuals during sleep, providing a foundation for future automated, video-based analysis of sleep movements that extends beyond conventional sensor-based methods.

We developed HypnoPose and tested five different model configurations. We pretrained each variant on a public body pose dataset and evaluated on a sleep pose dataset comprising 4419 annotated frames from 198 video segments depicting movement across 74 participants (10 RBD, 4 PD, 9 healthy controls, 51 referred for vPSG screening) recorded in clinical and home settings. We manually annotated each frame with 13 body joints, visibility flags, and head orientation. We evaluated model performance against state-of-the-art pose estimators using precision (mAP) and recall (mAR) metrics based on Object Keypoint Similarity (OKS).

HypnoPose achieved the highest performance (mAP: 0.088, AP@0.5: 0.326) in the sleep domain, doubling baseline HigherHRNet results (mAP: 0.041, AP@0.5: 0.165) and outperforming gold-standard architectures. It showed 40%-120% relative mAP improvement for occluded joints, enhancing detection of the head, shoulders, and elbows. Home recordings showed higher precision than clinic data (mAP 0.14 vs 0.07). Within clinic recordings, NREM stages outperform Wake (mAP 0.11-0.13 vs 0.06).

We present a proof-of-concept for detecting upper-body joint positions from sleep images, even when blankets occlude the person. Our method improves relative precision by 115% compared to standard models. While absolute performance remains modest, this work establishes a first step toward clinically applicable, video-based pose estimation during sleep. Future work should integrate contextual priors and expand annotated sleep datasets. Statement of Significance Video polysomnography (vPSG) is the gold standard for assessing sleep-related disorders. It is resource-intensive, requires specialist expertise for setup and interpretation, and may not capture typical sleep due to the presence of multiple sensors and the laboratory environment. In some populations, such as individuals with REM sleep behavior disorder (RBD), vPSG can be challenging, and clinically relevant behaviors may not occur during a single recording night. At-home recordings provide greater environmental validity but are often constrained by nonstandard camera positioning and suboptimal lighting. This study introduces a benchmark for automated, frame-wise upper-body pose estimation, to identify anatomical joint positions during sleep. While tools for detecting periodic limb movements and REM sleep without atonia already exist, there remains a critical need for objective, scalable methods to quantify complex movements seen in disorders such as RBD, NREM parasomnias, and sleep-related epilepsies. This work lays the foundation for future automated, noncontact movement analysis that complements existing vPSG workflows, reduces manual review time, and supports accessible monitoring of sleep behavior in both clinical and home environments.