Robust comparative evaluation of 15 natural language processing algorithms to positively identify patients with
Matt Stammers1,2, Markus Gwiggner3,4, Reza Nouraei2,5
1Department of Gastroenterology and The Southampton Emerging Therapies and Technologies (SETT) Centre, University Hospital Southampton NHS Foundation Trust, Southampton, UK m.stammers@soton.ac.uk.
Objective:
Natural language processing (NLP) can identify cohorts of patients with inflammatory bowel disease (IBD) from free text. However, limited sharing of code, models, and data sets continues to hinder progress. The aim of this study was to evaluate multiple open-source NLP models for identifying IBD cohorts, reporting on document-to-patient-level classification, while exploring explainability, generalisability, fairness and cost.
Methods:
15 algorithms were assessed, covering all types of NLP spanning over 50 years of NLP development. Rule-based (regular expressions, spaCy with negation), and vector-based (bag-of-words (BoW), term frequency inverse document frequency (TF IDF), word-2-vector), to transformers: (two sentence-based sBERT models, three bidirectional encoder representations from transformers (BERT) models (distilBERT, BioclinicalBERT, RoBERTa), and five large language models (LLMs): (Mistral-Instruct-v0.3-7B, M42-Health/Llama-v3-8B, Deepseek-R1-Distill-Qwen-v2.5-32B, Qwen-v3-32B, and Deepseek-R1-Distill-Llama-v3-70B). Models were comparatively evaluated based on full confusion matrices, time/environmental costs, fairness, and explainability.
Results:
A total of 9311 labelled documents were evaluated. The fine-tuned DistilBERT_IBD model achieved the best performance overall (micro F1: 93.54%), followed by sBERT-Base (micro F1: 93.05%); however, specificity was an issue for both: (67.80-64.41%) respectively. LLMs performed well, given that they had never seen the training data (micro F1: 86.47-92.20%), but were comparatively slow (18-300 hours) and expensive. Bias was a significant issue for all effective model types.
Conclusion:
NLP has undergone significant advancements over the last 50 years. LLMs appear likely to solve the problem of re-identifying patients with IBD from clinical free text sources in the future. Once cost, performance and bias issues are addressed, they and their successors are likely to become the primary method of data retrieval for clinical data warehousing.
More Related Videos
06:22Machine Learning-Based Cough Tone Classification: Diagnostic Exploration of Chronic Obstructive Pulmonary Disease and Respiratory Tract Infections
Published on: September 19, 2025
07:50A Metadata Extraction Approach for Clinical Case Reports to Enable Advanced Understanding of Biomedical Concepts
Published on: September 20, 2018
Related Concept Videos
01:30Inflammatory Bowel Disease III: Diagnostic Studies and Management I-Nutritional Therapy
Diagnostic studies
A colonoscopy is the definitive screening test, distinguishing ulcerative colitis from other colon diseases with similar symptoms. During a colonoscopy test, inflamed mucosa with exudate ulcerations can be observed, and biopsies are taken to determine the histologic characteristics of the...
01:30Inflammatory Bowel Disease II: Crohn's Disease
Inflammatory bowel disease, commonly known as IBD, refers to a collection of disorders that lead to persistent inflammation of the gastrointestinal tract. The two types of IBD are ulcerative colitis, which impacts the colon, and Crohn's disease, which can involve any part of the gastrointestinal segment.
Crohn's disease
Crohn's disease is a chronic, systemic inflammatory bowel disease (IBD) that predominantly affects the gastrointestinal tract. It is marked by...
01:21Inflammatory Bowel Disease V: Surgical Management
Here are some common surgical interventions for IBD:
01:17Chronic Bowel Disorders: Introduction
Irritable Bowel Syndrome (IBS) is a common disorder affecting the gastrointestinal tract. The distinctive feature is recurrent abdominal pain associated with altered bowel movements, manifesting as constipation, diarrhea, or fluctuating between both. The...
01:30Irritable Bowel Syndrome II: Clinical Features and Diagnostic Evaluation
Irritable Bowel Syndrome (IBS) is classified into subtypes based on the predominant bowel habits as determined by the Bristol Stool Form Scale (BSFS). The subtypes are:
01:27Inflammatory Bowel Disease I: Ulcerative Colitis
Inflammatory bowel disease, or IBD, encompasses a group of disorders characterized by chronic inflammation or ulceration of the gastrointestinal tract.
Risk Factors
The exact cause of IBD remains unclear, although it is believed to be due to a mix of genetic, environmental, microbial, and immune factors. Genetic factors are significant in determining susceptibility to IBD, with family history being a critical risk factor. Individuals with a first-degree relative who has IBD are at...