Evaluation of Open-Source Large Language Models for Metal-Organic Frameworks Research

Along with the development of machine learning, deep learning, and large language models (LLMs) such as GPT-4 (GPT: Generative Pre-Trained Transformer), artificial intelligence (AI) tools have been playing an increasingly important role in chemical and material research to facilitate the material screening and design. Despite the exciting progress of GPT-4 based AI research assistance, open-source LLMs have not gained much attention from the scientific community. This work primarily focused on metal-organic frameworks (MOFs) as a subdomain of chemistry and evaluated six top-rated open-source LLMs with a comprehensive set of tasks including MOFs knowledge, basic chemistry knowledge, in-depth chemistry knowledge, knowledge extraction, database reading, predicting material property, experiment design, computational scripts generation, guiding experiment, data analysis, and paper polishing, which covers the basic units of MOFs research. In general, these LLMs were capable of most of the tasks. Especially, Llama2-7B and ChatGLM2-6B were found to perform particularly well with moderate computational resources. Additionally, the performance of different parameter versions of the same model was compared, which revealed the superior performance of higher parameter versions.