ChatEndoscopist: A Domain-Specific Chatbot with Images for Gastrointestinal Diseases

Area of Science:

• Medical Informatics
• Artificial Intelligence in Medicine

Background:

• Large language models (LLMs) require domain-specific knowledge enhancement.
• Esophageal cancer research necessitates specialized information retrieval and understanding.

Purpose of the Study:

• To develop chatEndoscopist, a specialized chatbot for esophageal cancer knowledge.
• To evaluate chatEndoscopist's performance against a general-purpose LLM, ChatGPT.
• To assess the utility of incorporating images in medical chatbots.

Main Methods:

• Fine-tuning a BioMistral LLM with 50 medical literature documents.
• Developing chatEndoscopist with integrated image retrieval capabilities.
• Comparative analysis of chatEndoscopist and ChatGPT using precision, recall, F1 score, Type-Token Ratio, and Lexical Density.

Main Results:

• ChatGPT outperformed chatEndoscopist in precision (0.210 vs. 0.148), recall (0.323 vs. 0.049), and F1 score (0.266 vs. 0.099).
• ChatGPT exhibited superior lexical diversity (TTR: 0.772 vs. 0.717) and lexical density (0.813 vs. 0.781).
• Image retrieval was functional but primarily for query elucidation.

Conclusions:

• Limited fine-tuning data may have impacted chatEndoscopist's specialized accuracy.
• Further research is needed to balance domain-specific accuracy with linguistic flexibility in medical LLMs.
• Integrating images shows potential but requires further development for optimal impact.