Comparison of automated deep neural network against manual sleep stage scoring in clinical data
Hanrong Cheng1, Yifei Yang2, Jingshu Shi2
1Department of Sleep Medicine, Institute of Respiratory Diseases, Shenzhen People's Hospital, The Second Clinical Medical College of Jinan University, The First Affiliated Hospital of Southern University of Science and Technology, Shenzhen, 518020, China.
Objective:
To compare the accuracy and generalizability of an automated deep neural network and the Philip Sleepware G3™ Somnolyzer system (Somnolyzer) for sleep stage scoring using American Academy of Sleep Medicine (AASM) guidelines.
Methods:
Sleep recordings from 104 participants were analyzed by a convolutional neural network (CNN), the Somnolyzer and skillful technicians. Evaluation metrics were derived for different combinations of sleep stages. A further comparison between the Somnolyzer and the CNN model using a single-channel signal as input was also performed. Sleep recordings from 263 participants with a lower prevalence of OSA served as a cross-validation dataset to validate the generalizability of the CNN model.
Results:
The overall agreement between automated and manual scoring for sleep staging in 104 participants outperformed that of the Somnolyzer according to various metrics (accuracy: 81.81 % vs. 77.07 %; F1: 76.36 % vs. 73.80 %; Cohen's kappa: 0.7403 vs. 0.6848). The results showed that the left electrooculography (EOG) single-channel model had minor advantages over the Somnolyzer. In terms of consistency with manual sleep staging, the CNN model demonstrated superior performance in identifying more pronounced sleep transitions, particularly in the N2 stage and sleep latency metrics. Conversely, the Somnolyzer showed enhanced proficiency in the analysis of REM stages, notably in measuring REM latency. The accuracy in the cross-validation set of 263 participants was also above 80 %.
Conclusions:
The CNN-based automated deep neural network outperformed the Somnolyzer and is sufficiently accurate for sleep study analyses using the AASM classification criteria.
