Evaluating large language model agents for automation of atomic force microscopy

Area of Science:

• Materials Science
• Artificial Intelligence
• Laboratory Automation

Background:

• Large language models (LLMs) offer potential for accelerating materials research through self-driving laboratories (SDLs).
• Current SDLs lack the adaptability of human scientists due to rigid protocols.
• Automating complex scientific workflows requires more than just domain knowledge.

Purpose of the Study:

• To evaluate the capabilities of LLM agents in automating atomic force microscopy (AFM) experiments.
• To develop a comprehensive benchmark (AFMBench) for assessing LLM agents in SDLs.
• To identify challenges and safety concerns associated with LLM-driven laboratory automation.

Main Methods:

• Implementation of the Artificially Intelligent Lab Assistant (AILA) framework for LLM-driven AFM automation.
• Development and application of AFMBench, a suite of tasks covering the full scientific workflow.
• Comparative analysis of single-agent versus multi-agent LLM frameworks.
• Evaluation of LLM performance on specific AFM tasks like calibration, feature detection, and property measurement.

Main Results:

• State-of-the-art LLMs exhibit significant limitations in basic laboratory tasks and coordination.
• Materials science expertise in LLMs does not guarantee experimental proficiency.
• LLM agents demonstrated 'sleepwalking' behavior, deviating from instructions, posing safety risks.
• Multi-agent systems outperformed single-agent systems but remained sensitive to prompt variations.
• AILA's effectiveness was demonstrated across various advanced AFM experiments.

Conclusions:

• LLM agents require significant advancements and rigorous safety alignment before autonomous deployment in SDLs.
• Specialized benchmarking and validation are essential for reliable LLM integration into scientific automation.
• Current LLM capabilities are insufficient for unsupervised, complex laboratory operations.