A systematic assessment of large language models' knowledge of rare diseases: How much do large language models know...

Large language models (LLMs) perform well on general medical benchmarks, but their ability to reason about rare diseases (RDs) remains unclear. Rather than challenge LLMs to diagnose a limited number of cases that are unlikely to represent all RDs or RD-associated genes, we instead sought to comprehensively probe LLM understanding of RD-associated genes and phenotypes. We systematically evaluated six leading general-domain LLMs (GPT-4, Claude 3.7, Llama-3.3 70B, Gemma-2 27B, Llama-3.2, and Phi-4) for their ability to generate core phenotypic features and causal genes required to support reasoning for 10,892 Orphanet diseases. Outputs were mapped to Human Phenotype Ontology (HPO) terms and HGNC gene symbols and compared with curated references using set overlap, semantic similarity, and disease ranking via the likelihood ratio interpretation of clinical abnormality (LIRICAL) framework applied to 8,000 patient Phenopackets. LLM recall of curated RD knowledge was generally low, with gene associations retrieved more accurately than phenotypes. Commercial models, particularly GPT-4 and Claude, achieved over 60% recall for gene associations but struggled with precise phenotype recovery. Despite low exact overlaps, moderate semantic similarity scores indicated partial alignment with curated data. When used in LIRICAL, LLM-derived phenotypic profiles yielded ranking performance close to that of gold standard profiles, although direct diagnostic accuracy remained limited. Interestingly, convergent non-curated terms across models suggest potential for hypothesis generation. Current generalist LLMs lack the precision to replace curated RD knowledge bases but offer complementary, semantically relevant information. Our results support hybrid approaches that combine expert curation with selectively integrated LLM outputs to enhance and scale ontology-driven RD diagnostics.