Identifying and Extracting Rare Diseases and Their Phenotypes with Large Language Models

Area of Science:

• Computational biology
• Medical informatics
• Natural Language Processing

Background:

• Rare disease diagnosis and treatment rely heavily on accurate phenotyping.
• Disease phenotypes are often buried in unstructured clinical text, posing a challenge for automated extraction.
• Developing large annotated corpora for rare diseases is a significant bottleneck.

Purpose of the Study:

• This study is the first to investigate prompt learning using large language models (LLMs) for identifying and extracting rare disease phenotypes.
• The research explores the efficacy of prompt learning in zero-shot and few-shot settings for rare disease phenotyping.

Main Methods:

• A comparative analysis was performed between prompt learning with ChatGPT and fine-tuning with BioClinicalBERT.
• Novel prompts were engineered for ChatGPT to extract rare diseases, symptoms, and signs.
• A benchmark was established for performance evaluation, including in-depth error analysis.

Main Results:

• Fine-tuning BioClinicalBERT achieved a higher overall F1 score (0.689) compared to ChatGPT in zero-shot (0.472) and few-shot (0.610) settings.
• ChatGPT demonstrated superior accuracy for rare diseases and signs in the one-shot setting (F1 of 0.778 and 0.725, respectively).
• Conversational, sentence-based prompts outperformed structured lists in accuracy.

Conclusions:

• Prompt learning with ChatGPT has the potential to match or exceed BioClinicalBERT's performance in extracting rare diseases and signs with minimal annotated data.
• The accessibility of ChatGPT offers a viable alternative for rare disease phenotyping, reducing reliance on large annotated corpora.
• Critical evaluation of LLM outputs is essential for ensuring the accuracy of rare disease phenotyping.