A visual language model enabling intelligent nanomaterial scanning electron micrograph annotation.

Area of Science:

• Materials Science
• Nanotechnology
• Computer Science

Background:

• Data-driven approaches in materials science are powerful but hindered by the need for large labeled datasets.
• Manual annotation of Scanning Electron Microscopy (SEM) images for nanomaterials is complex and time-consuming.
• Automated pattern recognition for SEM images is crucial due to the scarcity of labeled data.

Purpose of the Study:

• To develop an automatic pattern recognition technology for SEM images of nanomaterials that does not rely on labeled data.
• To adapt existing Vision-Language Models (VLMs) for the specific domain of nanomaterials science.
• To reduce the dependency on large labeled datasets in AI-driven materials research.

Main Methods:

• Developed the Scanning Electron Microscopy Vision-Language Model (SEM-VLM) by adapting a general Vision-Language Model.
• Trained SEM-VLM using contrastive learning on SEM image-text pairs extracted from scientific literature.
• Employed ensemble vision-language alignment for zero-shot classification and activation mapping for interpretability.

Main Results:

• SEM-VLM outperformed general-domain models like CLIP and random baselines in cross-modal retrieval (Recall@10, Recall@50).
• Achieved high accuracy in zero-shot classification, surpassing CLIP.
• Demonstrated superior performance in few-shot settings compared to fully supervised models, using only 2.1% of training labels.
• Activation mapping provided interpretable localization of nanoscale features.

Conclusions:

• SEM-VLM offers a robust and interpretable solution for automated analysis of nanomaterial SEM images.
• The model significantly reduces the dependency on labeled datasets, enabling high-precision classification with minimal supervision.
• This multimodal framework advances AI applications in materials science, particularly for complex nanomaterial characterization.