Development of a Bariatric Surgery Specific Artificial Intelligence Large Language Model: BariatricSurgeryGPT

Area of Science:

• Artificial Intelligence
• Medical Informatics
• Surgical Research

Background:

• Commercially available large language models (LLMs) possess broad capabilities but lack clinical domain specificity.
• General LLMs exhibit limitations in reliability for specialized medical applications like bariatric surgery.
• There is a need for AI tools with enhanced accuracy and clinical relevance in bariatric surgery.

Purpose of the Study:

• To develop and evaluate BariatricSurgeryGPT, a fine-tuned LLM for bariatric surgery.
• To improve the accuracy and clinical relevance of LLM responses to bariatric surgery questions.
• To assess the performance of a domain-specific LLM against a baseline model.

Main Methods:

• Collected 8764 bariatric surgery research abstracts from PubMed (2020-2024).
• Preprocessed and tokenized abstracts to fine-tune a GPT-2 model using PyTorch and HuggingFace.
• Evaluated model performance using BLEU, METEOR, and ROUGE-1 scores on 20 clinical questions across nine temperature settings.

Main Results:

• BariatricSurgeryGPT showed consistent improvements over the baseline GPT-2 model.
• Achieved a 12.8% improvement in BLEU score (0.165 vs 0.147).
• Demonstrated significant gains in METEOR (8.2% improvement, 0.633) and ROUGE-1 (9.7% improvement, 0.267) scores.

Conclusions:

• BariatricSurgeryGPT is the first domain-specific LLM for bariatric surgery, proving the feasibility of specialty-specific AI.
• The fine-tuned model offers enhanced precision, recall, and semantic relevance for bariatric surgery content.
• Potential applications include surgical education, patient communication, and clinical decision support.