Automated identification of diagnostic labelling errors in medicine

Area of Science:

• Medical Informatics
• Health Services Research
• Clinical Decision Support

Background:

• Diagnostic error identification is challenging, often relying on subjective expert ratings.
• Routine healthcare data, coded using the International Classification of Diseases (ICD), presents an underutilized resource for quality assessment.

Purpose of the Study:

• To develop and validate a system for automatically identifying diagnostic labeling errors from ICD-coded diagnoses.
• To assess the system's performance against expert-based classifications.

Main Methods:

• A novel system was developed to calculate the distance between paired diagnoses within the ICD taxonomy.
• Validation involved comparing the system's output (index test) against expert classifications (reference standard) from three prior studies.
• Performance was evaluated using the area under the receiver operating characteristics curve (AUROC).

Main Results:

• The system demonstrated excellent performance, with overall AUROC values ranging from 0.821 to 0.837.
• Performance varied by algorithm, with the highest AUROC reaching 0.924 when analyzed per dataset type.
• The system's accuracy was robust and not significantly affected by the limitations of the reference standards.

Conclusions:

• The automated system effectively identifies diagnostic labeling errors in routine healthcare data.
• This approach overcomes the limitations of manual expert review, offering a scalable solution.
• Applicability is currently limited to cases with pairs of diagnoses where one is demonstrably superior to the other.