Fine-tuning large language models for chemical text mining

Area of Science:

• Computational chemistry
• Natural Language Processing
• Chemical Informatics

Background:

• Extracting knowledge from chemical literature is challenging due to complex language.
• Automated data acquisition is crucial for both experimental and computational chemists.
• Large Language Models (LLMs) show potential for chemical text mining.

Purpose of the Study:

• To explore the effectiveness of fine-tuned LLMs on intricate chemical text mining tasks.
• To compare fine-tuned LLMs against prompt-engineered models (ChatGPT, GPT-4) and other open-source LLMs.
• To assess the performance of LLMs with minimal annotated data.

Main Methods:

• Fine-tuning of various LLMs including ChatGPT (GPT-3.5-turbo), GPT-4, Mistral, Llama3, Llama2, T5, and BART.
• Evaluation on five chemical text mining tasks: compound entity recognition, reaction role labelling, MOF synthesis extraction, NMR data extraction, and reaction-to-action sequence conversion.
• Comparison of fine-tuned models against prompt-engineered models using limited annotated data.

Main Results:

• Fine-tuned ChatGPT models achieved high accuracy (69%-95%) across all evaluated tasks.
• Fine-tuned LLMs outperformed models using task-adaptive pre-training with larger in-domain datasets.
• Fine-tuned Mistral and Llama3 demonstrated competitive performance.
• Significant reduction in prompt engineering effort was observed.

Conclusions:

• Fine-tuned LLMs are highly effective for chemical knowledge extraction, even with minimal data.
• These models offer a versatile, robust, and low-code solution for automated data acquisition.
• Leveraging fine-tuned LLMs can revolutionize the field of chemical informatics.