Note-Level Phenotyping of Multiple-Sclerosis Notes by a Large Language Model Achieves near Human-Level Agreement.

Area of Science:

• Medical Informatics
• Natural Language Processing
• Clinical Research

Background:

• Clinical phenotyping from electronic health records (EHRs) typically uses complex multi-stage pipelines.
• Large language models (LLMs) offer potential for direct document-level abstraction of phenotype features.
• Evaluating LLMs for direct note-level phenotyping is crucial for advancing EHR data utilization.

Purpose of the Study:

• To assess GPT-5.2's capability for note-level multiple sclerosis (MS) phenotyping.
• To compare GPT-5.2's performance against human annotators and other automated methods.
• To determine the feasibility of direct LLM-based abstraction for EHR phenotyping.

Main Methods:

• Analysis of 100 de-identified MS neurology progress notes.
• Annotation of 17 predefined neurological phenotype categories by two human annotators.
• Evaluation of GPT-5.2 in a zero-shot setting and comparison with Llama-3.1 8B, Doc2Hpo, ClinPhen, PhenoSnap, and BioClinical ModernBERT.

Main Results:

• GPT-5.2 achieved strong automated performance (macro-F1 0.801), approaching human annotator levels.
• GPT-5.2 demonstrated the lowest false-negative rate but a higher false-positive rate than humans.
• Llama-3.1 8B performed competitively; HPO-based tools and BioClinical ModernBERT showed lower performance.

Conclusions:

• GPT-5.2 demonstrated near-human performance in recognizing MS phenotype categories from narrative notes.
• Direct note-level abstraction using LLMs presents a scalable strategy for EHR phenotyping.
• This approach could significantly enhance research and population health initiatives using large EHR corpora.