Evaluation of Open-Source Large Language Models for Metal-Organic Frameworks Research

Area of Science:

• Chemistry
• Materials Science
• Artificial Intelligence

Background:

• Artificial intelligence (AI) tools, including deep learning and large language models (LLMs) like GPT-4, are increasingly vital in chemical and material research for screening and design.
• Open-source LLMs, despite their potential, have received limited attention within the scientific community compared to proprietary models.

Purpose of the Study:

• To evaluate the capabilities of six leading open-source LLMs for various tasks within metal-organic frameworks (MOFs) research.
• To assess the performance of different parameter versions of the same LLM, identifying optimal models for scientific applications.

Main Methods:

• Comprehensive evaluation of six open-source LLMs on tasks including knowledge retrieval, property prediction, experiment design, and data analysis specific to MOFs.
• Comparative analysis of model performance based on varying parameter counts.

Main Results:

• Open-source LLMs generally demonstrated proficiency across a wide range of MOFs research tasks.
• Llama2-7B and ChatGLM2-6B emerged as top performers, requiring moderate computational resources.
• Higher parameter versions of LLMs consistently outperformed lower parameter versions.

Conclusions:

• Open-source LLMs are viable tools for accelerating research in specialized chemical domains like MOFs.
• Llama2-7B and ChatGLM2-6B offer efficient and effective AI assistance for chemists and material scientists.
• Model scale (parameter count) is a significant factor influencing the performance of LLMs in scientific research.