Benchmarking proprietary and open-source language and vision-language models for gastroenterology clinical reasoning

Area of Science:

• Medical Artificial Intelligence
• Gastroenterology Applications
• Natural Language Processing

Background:

• The integration of advanced AI, including large language models (LLMs) and vision-language models (VLMs), is rapidly expanding across medical disciplines.
• Assessing the performance of these models in specialized fields like gastroenterology is crucial for understanding their potential clinical utility.

Purpose of the Study:

• To evaluate the effectiveness and accuracy of various proprietary and open-source LLMs and VLMs in a gastroenterology context.
• To compare model performance across different interfaces, computing environments, and quantization levels.

Main Methods:

• Utilized board-style, multiple-choice questions relevant to gastroenterology to test LLMs and VLMs (e.g., GPT, Claude, Gemini, Mistral, Llama, Mixtral, Phi, Qwen).
• Assessed performance under varying conditions, including different compression (quantization) levels for open-source models.
• Investigated the impact of image captions (human-generated, original, LLM-generated) on VLM performance for image-based questions.

Main Results:

• Proprietary models o1-preview (82.0%) and Claude3.5-Sonnet (74.0%) achieved the highest accuracy.
• Top open-source models Llama3.3-70b (65.7%) and Qwen-2.5-72b (61.0%) were outperformed by proprietary models.
• Small, quantized open-source models (8-bit Llama 3.2-11b, 6-bit Phi3-14b) showed performance comparable to their full-precision versions.
• Vision-language model accuracy increased with human-generated captions but decreased with LLM-generated captions for image-based questions.

Conclusions:

• Proprietary LLMs and VLMs demonstrate superior performance in gastroenterology assessments compared to current open-source models.
• Quantization offers a viable method for reducing the computational requirements of open-source models without significant accuracy loss.
• The quality of image captions critically influences VLM performance, highlighting the need for careful data preparation in clinical applications.