An adaptive, variable-length test for certifying electroencephalographic spike-detection skill: A retrospective...

Determining whether an individual reader has sufficient electroencephalographic (EEG) spike-detection skill is challenging due to limited expert agreement. We evaluated whether modeling skill in interpretable signal-detection terms-discrimination ability separate from response tendency-could power an adaptive, variable-length test that can reliably and efficiently assess reader competence.

We analyzed a multirater dataset containing 20 521 EEG segments and 2574 rater sessions. Case signal was estimated from multirater labels, and each reader was modeled using signal-detection parameters: discrimination skill $\sigma$ and response bias $t$ . Among 368 calibration-eligible raters across five skill tiers, we used a stratified 50/50 train-validation split to select a certification threshold ${\sigma }^{\star }$ in training and to apply it in validation. We replayed an adaptive test on each rater's historical responses, using sequential stopping to issue pass, fail, or retest decisions.

The training-derived threshold was ${\sigma }^{\star }=.815$ (95% bootstrap confidence interval = $.663$ - $1.018$ , Youden $J=.828$ ). In validation, pass rates preserved the expected tier ordering: experts 100% ( $14/14$ ), experienced 100% ( $4/4$ ), novices 14% ( $1/7$ ), residents 0% ( $0/11$ ), and crowd raters 7% ( $11/150$ ). In the 81-rater adaptive replay, clear failures stopped near 20 trials, and passing experts required a median of 95 trials. Split-half replay showed intraclass correlation coefficient $\left(3,1\right)=.87$ for $\hat{\sigma }$ at stopping; there were no hard pass-fail reversals among raters whose two halves both reached decisive verdicts.

A signal-detection framework quantified spike-identification skill in EEG readers, and adaptive testing estimated this skill reliably with fewer than 100 short EEG segments. This approach could provide a practical, scalable method for assessing EEG reading competence.