MicroscopyGPT: Generating Atomic-Structure Captions from Microscopy Images of 2D Materials with Vision-Language...

Area of Science:

• Materials Science
• Computational Materials Science
• Nanotechnology

Background:

• Determining complete atomic structures from microscopy images is a significant challenge.
• Automated methods are needed to accelerate materials discovery and analysis.

Purpose of the Study:

• To introduce MicroscopyGPT, a novel vision-language model (VLM) for predicting atomic structures from scanning transmission electron microscopy (STEM) images.
• To demonstrate the capability of VLMs in translating microscopy data into detailed crystallographic information.

Main Methods:

• Developed MicroscopyGPT, a VLM based on multimodal generative pretrained transformers.
• Trained the model on a diverse dataset of simulated STEM images for approximately 5000 2D materials.
• Fine-tuned the 11-billion-parameter LLaMA model for efficient training.

Main Results:

• MicroscopyGPT accurately predicts full atomic configurations, including lattice parameters, element types, and atomic coordinates from STEM images.
• The model successfully learns complex mappings between image features and crystallographic representations.

Conclusions:

• MicroscopyGPT offers a powerful automated approach for atomic structure reconstruction from microscopy data.
• This VLM has broad implications for accelerating materials discovery, nanotechnology, and catalysis research.