Interpretable Fine-tuned Large Language Models Facilitate Making Genetic Test Decisions for Rare Diseases | JoVE Visualize

Area of Science:

Medical Informatics
Artificial Intelligence in Healthcare
Genomic Medicine

Background:

Clinical decision-making for rare diseases often relies on expert interpretation of complex guidelines, such as those from the American College of Medical Genetics and Genomics (ACMG).
Standardizing and implementing these guidelines for genetic test selection (gene panels vs. whole exome/genome sequencing) presents challenges.
Traditional machine learning models struggle with the interpretability required for clinical guideline application.

Purpose of the Study:

To develop and evaluate an approach using large language models (LLMs) to interpret clinical guidelines and assist in selecting appropriate genetic diagnostic modalities for rare diseases.
To enhance the interpretability of AI models in clinical decision support by mimicking clinician reasoning patterns through chain-of-thought (CoT).

Main Methods:

Developed RareDAI, an integrative approach analyzing heterogeneous clinical data (unstructured notes, Phecodes).
Fine-tuned Llama 3.1 and Qwen 3 models using domain-specific questions to generate structured chain-of-thought (CoT) outputs.
Employed a novel self-distillation fine-tuning (SDFT) approach to refine LLM reasoning for interpretable recommendations.

Main Results:

RareDAI demonstrated superior performance compared to traditional supervised fine-tuning and base LLMs (Llama 3.1, GPT-4).
Improvements of 10-20% in accuracy, precision, recall, and F1-score were observed across in-house and external datasets.
The fine-tuned models produced interpretable reasoning chains prior to providing recommendations.

Conclusions:

Large language models, when fine-tuned with chain-of-thought and self-distillation, can effectively interpret clinical guidelines for rare disease diagnosis.
RareDAI significantly enhances the selection of diagnostic modalities, outperforming existing methods and aiding clinical decision-making.
This approach offers a promising pathway for integrating AI into genomic medicine to improve diagnostic efficiency and accuracy.