Quantifying Interpretation Reproducibility in Vision Transformer Models with TAVAC.

Area of Science:

• Artificial Intelligence
• Biomedical Imaging
• Digital Pathology

Background:

• Deep learning, particularly Vision Transformer (ViT) models, shows promise for extracting diagnostic features from biomedical images, surpassing Convolutional Neural Networks (CNNs) in image classification and interpretability.
• However, limited annotated datasets can cause ViT models to overfit, leading to unreliable predictions and compromised model interpretation, especially in digital pathology applications.

Approach:

• Introduced a novel metric, Training Attention and Validation Attention Consistency (TAVAC), to evaluate ViT model overfitting and quantify interpretation reproducibility.
• TAVAC compares high-attention regions between training and testing phases to assess the consistency of model interpretation.

Key Points:

• Tested on diverse image classification and histological datasets, including breast cancer images.
• Overfitted models consistently showed significantly lower TAVAC scores compared to well-fitted models.
• TAVAC provides a fine-grained, quantitative measure of interpretation generalization, surpassing traditional metrics like accuracy.

Conclusions:

• TAVAC establishes a new standard for evaluating deep learning model interpretation performance, particularly for biomedical images.
• The metric enhances the monitoring of interpretative reproducibility at pixel-resolution, aiding basic research and disease mechanism discovery.
• TAVAC helps determine the significance of attention map regions, crucial for transparent and reliable AI in diagnostics and research.