Process and outcome for international reliability in sleep scoring

The aim was to evaluate the inter-rater reliability in scoring sleep stages in two sleep labs in Berlin Germany and Beijing China.

The subjects consist of polysomnography (PSGs) from 15 subjects in a German sleep laboratory, with 7 mild to moderate sleep apnea hypopnea syndrome (SAHS) patients and 8 healthy controls, and PSGs from 15 narcolepsy patients in a Chinese sleep laboratory. Five experienced technologists including two Chinese and three Germans without common training scored the PSGs following the 2007 AASM manual except the EEG signals included only two EEG leads (C3/A2 and C4/A1). Differences in inter-scorer agreement were analyzed based on epoch-by-epoch comparison by means of Cohen's κ, and quantitative sleep parameters by means of intra-class correlation coefficients.

Inter-laboratory epoch-by-epoch agreement comparison between scorers from the two countries yielded a moderate agreement with a mean κ value of 0.57 for controls, 0.58 for SAHS, and 0.54 for narcolepsy. When compared with controls, the inter-scoring agreement is higher for wake and N3 stage scoring in SAHS and N1 and N3 scoring in narcolepsy (p < 0.05). The only sleep stage with lower scoring agreement in both SAHS (κ 0.69 vs. 0.79, p = 0.034) and narcolepsy (0.66 vs 0.79, p = 0.022) was stage REM. Inter-laboratory comparisons showed that the most common combinations of deviating scorings were N1 and N2, N2 and N3, and N1 and wake. A 6.5 % deviating scoring rate of wake and REM and a 13.4 % deviating scoring rate of N1 and REM indicated that inter-laboratory scoring in narcolepsy was about twice as in SAHS and controls confused. This was further confirmed by agreement analysis of quantitative parameters using intra-class correlation coefficients ICC(2,1) indicating REM sleep scoring agreement was lower in narcolepsy than in controls (p < 0.05).

Low REM stage scoring agreement exists for narcoleptics and SAHS, indicating the necessity to study sleep stage scoring agreement for a specific sleep disorder. Intensive training is needed for the scoring of sleep in international multiple center studies to improve the scoring agreement.