Inter-rater agreement for visual discrimination of phasic and tonic electromyographic activity in sleep

The objective of this study was to determine the confidence of expert raters in discriminating phasic and tonic electromyographic (EMG) activity. We undertook this study because we suspected that even expert scorers may disagree on whether a given EMG segment contained phasic activity, tonic activity, or both.

Six individuals holding either Fellowship status in the American Academy of Sleep Medicine or Board Certification in Sleep Medicine with at least 5 years experience in interpreting polysomnography visually examined 60 segments containing EMG activity. Raters determined their relative confidence that each segment contained phasic and tonic activity by noting whether they were highly certain or somewhat certain that the segment contained such activity or somewhat certain or highly certain that each segment did not contain such activity. Every segment was rated by every rater twice, once for phasic and once for tonic activity.

Substantial differences among raters existed in certainty regarding presence/absence of both phasic and tonic activity, although raters agreed on segments far above chance. Consensus was higher on certainty regarding presence of phasic, relative to tonic, activity.

These findings indicate the limitations of visual analyses for discriminating abnormal muscle activity during sleep. Conversely, when expert judgments are combined with digitized measurements of EMG activity in sleep (e.g. REM atonia index), some allowance must be made for the unique contribution of visual analyses to such judgments, most notably for short duration EMG signals. These results may have relevance for polysomnographic interpretation in suspected synucleinopathies.