A hybrid framework with large language models for rare disease phenotyping.

Area of Science:

• Medical Informatics
• Computational Linguistics
• Rare Disease Research

Background:

• Rare diseases present diagnostic and treatment challenges due to low prevalence and varied symptoms.
• Unstructured clinical notes are rich in diagnostic information but difficult to analyze manually.
• Automated methods are needed to efficiently and accurately identify rare diseases from clinical text.

Purpose of the Study:

• To develop and evaluate a hybrid framework for enhanced rare disease identification from unstructured clinical reports.
• To combine dictionary-based NLP tools with large language models (LLMs) for improved accuracy.
• To leverage existing ontologies for comprehensive rare disease vocabulary creation.

Main Methods:

• Integrated Orphanet Rare Disease Ontology (ORDO) and Unified Medical Language System (UMLS) for a rare disease vocabulary.
• Utilized SemEHR, a dictionary-based NLP tool, for initial rare disease mention extraction.
• Employed various LLMs (LLaMA3, Phi3-mini, OpenBioLLM, BioMistral) with different prompting strategies (zero-shot, few-shot, knowledge-augmented generation).

Main Results:

• The hybrid approach outperformed traditional NLP and standalone LLMs in rare disease identification.
• LLaMA3 and Phi3-mini achieved the highest F1 scores, with few-shot prompting (1-3 examples) being most effective.
• The method identified numerous potential rare disease cases missed by structured diagnostic records.

Conclusions:

• The hybrid NLP-LLM approach shows significant promise for improving rare disease identification from clinical notes.
• This method can uncover previously unrecognized rare disease cases, aiding early diagnosis.
• Further research should focus on ontology mapping, overlapping case identification, and clinical integration.