Comparing three natural language processing methods for the automatic identification of epilepsy patients from French...

Manual review of clinical notes by experts remains the reference standard for identifying patients with epilepsy in health databases. However, this process is labor-intensive and time-consuming due to the unstructured nature of text. Prior studies have shown the potential of natural language processing for automated phenotyping. We aim to develop and validate algorithms capable of identifying patients with epilepsy based on a set of clinical notes.

A population of 109 448 patients was selected from the Assistance Publique-Hôpitaux de Paris (AP-HP) Clinical Data Warehouse (CDW) (38 hospitals in Paris, France) based on the presence of an International Classification of Diseases, Tenth Revision (ICD-10) diagnostic code related to epilepsy (G40/G41) or mimicking disorders (R53/R55/R56), or the mention of at least one antiseizure medication in their medical chart. From this pre-screened population, 6733 sentences (from 2700 patients) were labeled as indicative or not indicative of epilepsy, and 3000 patients were selected randomly for manual review by a neurologist. We compared a "basic" keyword-based method, a rule-based method, and a pretrained language model for identifying epilepsy-related sentences and classifying patients with epilepsy. We reported the F1 score of each method.

At the sentence level, the pretrained language model reached the highest F1 score of .95 (95% confidence interval [CI]: .95-.96) outperforming the rule-based method .87 (95% CI: .86-.88) and the basic method .81 (95% CI: .80-.81). At the patient level, the pretrained language model also achieved the best F1 score .95 (95% CI: .94-.96) compared to the rule-based method .93 (95% CI: .91-.94) and the basic method .82 (95% CI: .81-.84).

Both the rule-based and the pretrained language models achieved high performance. These algorithms can automatically identify patients with epilepsy from unstructured clinical notes in French data warehouses, supporting large-scale phenotyping and the detection of epilepsy as a comorbidity.