Related Experiment Video
Updated: Jun 24, 2026
05:51
Assessing the Accuracy of Fitness Smartwatch Data for Cardiovascular and Physical Activity Monitoring: A Validation Study in Digital Health
Published on: February 21, 2025
What matters beyond model choice for wearable sleep staging? How personalization, evaluation choices, and
Eric Canton1, Franco Tavella1, Christopher Drake2
1Arcascope Inc., Arlington, VA, United States.
Sleep Advances : a Journal of the Sleep Research Society
|June 23, 2026
Summary
Dataset characteristics significantly impact wearable sleep classification performance, not just model choice. Including individuals with sleep apnea in training data improves results for suspected sleep disorder datasets. Researchers should use benchmark datasets and assess class separability.
Area of Science:
- Biomedical Engineering
- Sleep Science
- Machine Learning
Background:
- Improving sleep classification accuracy from wearable devices is crucial for consumer sleep technology.
- Current research often focuses on model selection, neglecting the impact of dataset characteristics.
- Lack of standardized benchmark datasets hinders the evaluation of factors contributing to performance gains.
Purpose of the Study:
- To investigate how dataset characteristics influence sleep classification performance, independent of model choice.
- To evaluate the impact of training and testing set composition on the performance of neural network models for sleep classification.
- To quantify the intrinsic ease of classification within different sleep datasets.
Main Methods:
- Collected the SleepAccel-Clinical dataset, including Apple Watch acceleration and polysomnography from 28 individuals with sleep apnea.
- Trained neural network models on SleepAccel-Clinical, SleepAccel (healthy individuals), and DREAMT (suspected sleep disorders).
- Utilized an Easy To Classify (ETC) wake model to quantify intrinsic class separability and compared its performance (AUROC) across datasets.
Main Results:
- The ETC model's performance significantly correlated with overall model performance across various datasets and architectures (p≪0.01).
- Incorporating individuals with obstructive sleep apnea (OSA) into the training data enhanced classification performance on datasets with suspected sleep disorders.
- Dataset characteristics, including the presence of ETC wake epochs, substantially influenced performance metrics.
Conclusions:
- Sleep classification performance is heavily influenced by training and testing dataset properties, not solely by model architecture.
- The Easy To Classify (ETC) wake model's presence can artificially inflate performance metrics.
- Researchers are encouraged to benchmark models using widely available datasets and to quantify intrinsic class separability when introducing new datasets.
