Natural language processing improves reliable identification of COVID-19 compared to diagnostic codes alone

Area of Science:

• Health Informatics
• Medical Record Analysis
• Epidemiological Research

Background:

• Observational studies on COVID-19 frequently utilize diagnostic codes for patient classification.
• The accuracy of these codes and potential for differential misclassification across diverse patient demographics remain largely unexamined.
• Understanding these limitations is crucial for reliable real-world evidence generation.

Purpose of the Study:

• To evaluate the accuracy of diagnostic codes for identifying COVID-19 patients.
• To investigate age, race, and ethnicity as predictors of differential misclassification.
• To compare the performance of diagnostic codes against natural language processing (NLP) classifiers.

Main Methods:

• Proof-of-concept study using two primary care cohorts from the American Family Cohort.
• Comparison of ICD-10 diagnostic codes against NLP classifiers trained on physician-assessed clinical notes.
• Three NLP models (tree-based, recurrent neural network, transformer-based) were trained and tested.

Main Results:

• Only 63% of likely COVID-19 patients had a documented ICD-10 code.
• Code sensitivity was lower in older patients (60.6% for 75+) and Black patients (58.5%).
• A tree-based NLP classifier achieved an AUC of 0.92 but showed decreased accuracy in older patients and required frequent retraining.

Conclusions:

• Diagnostic codes exhibit significant limitations in accurately identifying COVID-19 patients, with notable disparities across age, race, and ethnicity.
• NLP holds potential for improving cohort identification but necessitates continuous retraining to maintain performance.
• Future research should focus on robust methods for accurate patient classification in observational studies.