Achieving high inter-rater reliability in establishing data labels: a retrospective chart review study

Area of Science:

• Medical Informatics
• Machine Learning Data Quality

Background:

• Machine learning (ML) model accuracy in medical research relies on precise labeled data.
• This study evaluated inter-rater reliability (IRR) during retrospective electronic medical chart reviews.
• The goal was to generate high-quality labeled data for comorbidities and adverse events (AEs).

Purpose of the Study:

• To assess the inter-rater reliability (IRR) in a retrospective electronic medical chart review.
• To establish a methodology for creating high-quality labeled data for machine learning applications.
• To ensure consistency and accuracy in identifying comorbidities and adverse events (AEs).

Main Methods:

• Six registered nurses reviewed 70 patient charts, extracting data on 20 comorbidities and 18 AEs.
• Four iterative training rounds were conducted to enhance reviewer accuracy and consensus.
• Weighted Kappa coefficients were calculated to measure inter-rater reliability (IRR).

Main Results:

• Overall agreement, measured by Conger's Kappa, was 0.80 (95% CI: 0.78-0.82).
• Inter-rater reliability (IRR) scores remained high throughout the review process, ranging from 0.70 to 0.87.
• Consistent high agreement indicates the effectiveness of the training and review protocol.

Conclusions:

• A detailed chart review manual and structured training program led to high agreement among reviewers.
• This methodology provides a reliable foundation for generating high-quality labeled medical data.
• Accurate labeled data is essential for developing robust and effective machine learning algorithms in healthcare.