Scalable incident detection via natural language processing and probabilistic language models

Area of Science:

• Medical Informatics
• Natural Language Processing
• Pharmacovigilance

Background:

• Post-marketing safety surveillance is crucial for detecting clinical events.
• Current methods using structured data have limitations in precision and completeness.
• Natural Language Processing (NLP) offers potential for analyzing unstructured clinical text.

Purpose of the Study:

• To develop and validate a novel incident phenotyping approach using unstructured clinical textual data.
• To assess the generalizability of the approach across different phenotypes (suicide attempt, sleep-related behaviors).
• To evaluate the performance of the phenotyping model, including potential racial disparities.

Main Methods:

• Developed a novel phenotyping approach based on a validated method (PheRe) for analyzing entire healthcare records.
• Utilized unstructured clinical textual data, agnostic to Electronic Health Record (EHR) and note type.
• Validated the approach on large datasets for suicide attempt (89,428 records) and sleep-related behaviors (35,863 records) using silver and gold standards.

Main Results:

• Achieved an Area Under the Precision-Recall Curve (AUPR) of ~0.77 for suicide attempt phenotyping.
• Observed an AUPR of ~0.31 for sleep-related behaviors phenotyping.
• Identified performance differences across phenotypes and by coded race, highlighting the need for algorithmovigilance and debiasing.

Conclusions:

• The developed NLP-based phenotyping approach is a scalable method for identifying clinical events from unstructured text.
• The model demonstrates generalizability but requires careful validation and bias assessment before implementation in healthcare AI.
• Further research is needed to address performance disparities and ensure equitable application of these AI tools.