Inter-expert and intra-expert reliability in sleep spindle scoring.
Sabrina L Wendt1, Peter Welinder2, Helge B D Sorensen3
1Center for Sleep Science and Medicine, Stanford University, Palo Alto, CA, United States; Danish Center for Sleep Medicine, Glostrup University Hospital, DK-2600 Glostrup, Denmark.
Summary
Reliable sleep spindle scoring requires 2-3 experts for substantial agreement and 4+ for almost perfect agreement. This research quantifies expert consensus in sleep spindle scoring for improved sleep staging accuracy.
Area of Science:
- Neuroscience
- Sleep Medicine
- Biomedical Engineering
Background:
- Sleep spindles are crucial for sleep staging and are implicated in cognitive functions, aging, and neurological disorders.
- Accurate sleep spindle scoring is essential for understanding sleep architecture and its clinical relevance.
Purpose of the Study:
- To assess inter-expert and intra-expert agreement in sleep spindle scoring.
- To determine the number of experts required for reliable sleep spindle scoring datasets.
Main Methods:
- Analysis of 400 sleep EEG segments from 110 individuals.
- Scoring of sleep spindles by 24 Registered Polysomnographic Technologists (RPSGTs) at electrode C3-M2.
- Calculation of agreement using F1-scores, Cohen's kappa (κ), and intra-class correlation coefficient (ICC).
Main Results:
- Average intra-expert agreement: F1-score 72±7%, κ 0.66±0.07.
- Average inter-expert agreement: F1-score 61±6%, κ 0.52±0.07.
- Higher reliability for spindle amplitude/frequency than duration; qualitative confidence scores improve scoring reliability.
Conclusions:
- 2-3 experts yield 'substantial' reliability (κ: 0.61-0.8) for sleep spindle scoring datasets.
- 4+ experts are needed for 'almost perfect' reliability (κ: 0.81-1).
- Expert consensus is key to building reliable sleep spindle datasets for research and clinical applications.
