Preclinical HistoBench: A Pilot Benchmark Dataset for Evaluating Large Language Models on Preclinical...

Area of Science:

• * Preclinical histopathology
• * Artificial intelligence in pathology
• * Large language model (LLM) evaluation

Background:

• * Lack of standardized benchmarks for assessing LLMs in preclinical histopathology.
• * Need for multi-dimensional classification capabilities in AI models for pathology.
• * Development of a pilot dataset for LLM evaluation in this domain.

Purpose of the Study:

• * To create and present a benchmark dataset for evaluating LLM performance on histological samples.
• * To assess LLMs on multi-dimensional classification tasks including species, organ, staining, and preparation type.
• * To address the need for standardized evaluation metrics in AI-driven histopathology.

Main Methods:

• * Evaluation of three LLMs (GPT-4.1, GPT-4o-mini, Llama 3.2) on 378 preclinical histological samples.
• * Four classification dimensions: species (mouse, rabbit, rat), organ, staining method, and preparation type (frozen vs. paraffin-embedded).
• * Performance assessment using sensitivity, specificity, and confusion matrix analysis.

Main Results:

• * Substantial variation in LLM performance across tasks and high sensitivity to class imbalance.
• * GPT-4.1 showed balanced preparation type classification; Llama 3.2 struggled with paraffin samples.
• * Llama 3.2 identified all species but had poor mouse recognition; GPT-4.1 excelled in mouse identification.
• * Llama 3.2 demonstrated high staining classification performance; GPT-4o-mini achieved perfect H&E recognition.

Conclusions:

• * Current LLMs exhibit variable performance in histological classification, highly sensitive to class imbalance.
• * LLMs are not suitable for standalone diagnostic use in histopathology.
• * LLMs show potential as screening tools in research settings with human oversight.