Related Experiment Video
Updated: Jun 26, 2026
07:11
Functional Magnetic Resonance Imaging (fMRI) of the Visual Cortex with Wide-View Retinotopic Stimulation
Published on: December 8, 2023
Vision language models for scientific image analysis: an evaluation highlighting opportunities and challenges
Prateek Verma1, Minh-Hao Van1, Xintao Wu1
1Department of Electrical Engineering and Computer Science, University of Arkansas, Fayetteville, AR USA.
Npj Computational Materials
|June 25, 2026
Summary
Vision language models (VLMs) show promise for analyzing scientific microscopy images in tasks like classification and segmentation. While not yet expert-level, models like ChatGPT and Gemini demonstrate improved comprehension and segmentation capabilities.
Area of Science:
- Scientific image analysis
- Microscopy
- Artificial intelligence in science
Background:
- Vision language models (VLMs) like ChatGPT, Gemini, Llama, and LLaVA excel at processing visual and textual data.
- The Segment Anything Model (SAM) demonstrates advanced image segmentation.
- Microscopy images are crucial in biology, medicine, and materials science.
Purpose of the Study:
- To evaluate the performance of advanced VLMs (ChatGPT-5, Gemini-2.5Pro, Llama-3.2V, LLaVA-1.5) and SAM (SAM-2) on microscopy image analysis tasks.
- Assess capabilities in classification, segmentation, counting, and visual question answering (VQA).
Main Methods:
- Utilized microscopy images for evaluation.
- Tested models including ChatGPT-5, Gemini-2.5Pro, Llama-3.2V, LLaVA-1.5, and SAM-2.
- Focused on tasks: classification, segmentation, counting, and VQA.
Main Results:
- ChatGPT and Gemini showed strong performance in image comprehension.
- SAM demonstrated excellent object isolation capabilities.
- All models showed improvement over previous versions but did not reach domain expert accuracy, especially with complex image artifacts.
Conclusions:
- VLMs show significant potential for advancing scientific image analysis.
- Further development is needed to achieve expert-level performance on complex microscopy data.
- These models offer a promising foundation for future scientific discovery tools.
