Best-case kappa scores calculated retrospectively from EEG report databases

Area of Science:

• Neurology
• Medical Informatics

Background:

• Interrater reliability in electroencephalography (EEG) is crucial for accurate diagnosis.
• The traditional kappa (κ) score for assessing interrater reliability is time-consuming, requiring repeated readings of the same EEG studies.

Purpose of the Study:

• To introduce a novel, retrospective method for calculating the best-case kappa score (κBEST).
• To assess interrater reliability among EEG readers more efficiently.

Main Methods:

• Utilized one year of EEG reports from four adult readers.
• Employed SQL queries for EEG finding extraction and logistic regression models.
• Derived the κBEST statistic from logistic regression coefficients, incorporating patient age, location acuity, and reader influence.

Main Results:

• Older patient age and higher location acuity correlated with reduced sleep and increased diffuse abnormalities.
• Individual EEG reader performance significantly impacted between-reader κBEST scores.
• Within-reader κBEST scores were consistently higher than between-reader scores, indicating good internal consistency.

Conclusions:

• The κBEST metric effectively quantifies interrater reliability differences retrospectively across multiple EEG readers and reports.
• This method is generalizable to other reporting domains like pathology and radiology.
• κBEST can serve as a valuable tool for targeted quality control in medical reporting.