OmniPathoVQA: Benchmarking pathology vision-language models with Encyclopedia-scale knowledge.

Area of Science:

• Artificial Intelligence in Medicine
• Computational Pathology
• Medical Imaging Analysis

Background:

• Pathology vision-language models (VLMs) show potential for clinical decision support.
• Current evaluation benchmarks lack clinical relevance, hindering real-world deployment.
• Existing benchmarks suffer from poor vision-language alignment and limited disease scope.

Purpose of the Study:

• Introduce OmniPathoVQA, a comprehensive pathology visual question answering (VQA) benchmark.
• Establish a rigorous standard for evaluating the clinical-level performance of pathology VLMs.
• Facilitate the development of advanced clinical decision support systems in pathology.

Main Methods:

• Developed OmniPathoVQA covering all human anatomical systems and thousands of diseases.
• Leveraged pathology educational materials for fine-grained image-knowledge linking.
• Designed challenging VQA questions based on microanatomic features for deep reasoning assessment.

Main Results:

• Evaluated eighteen VLMs, noting superior performance of closed-source models on general tasks.
• Observed a significant performance drop in closed-source models on difficult, knowledge-intensive questions.
• Identified strong general reasoning ability as critical for VLM fine-tuning and pathology interpretation.

Conclusions:

• OmniPathoVQA provides a robust framework for assessing high-level reasoning in pathology VLMs.
• The benchmark highlights the need for improved reasoning capabilities in current VLMs for clinical utility.
• This work guides future development towards more clinically applicable pathology VLMs.