Benchmarking Large Language Models for Polymer Property Predictions

Machine learning and artificial intelligence have revolutionized polymer science by enhancing the ability to rapidly predict key polymer properties and enabling generative design. The utilization of large language models (LLMs) in polymer informatics may offers additional opportunities for advancement. Unlike traditional methods that depend on large labeled datasets, hand-crafted representations of the materials, and complex feature engineering, LLM-based methods utilize natural language inputs via a transfer learning process and eliminate the need for complex representation and fingerprinting, thus significantly simplifying the training process. In this study, we fine-tune general-purpose LLMs-open-source Llama-3-8B and commercial GPT-3.5-on a curated dataset of 11,740 entries to predict key thermal properties: glass transition ( $T_g$ ), melting ( $T_m$ ), and decomposition ( $T_d$ ) temperatures. Using parameter-efficient fine-tuning and hyperparameter optimization, we benchmark these models against traditional fingerprinting-based approaches including Polymer Genome, polyGNN, and polyBERT, under both single-task (ST) and multi-task (MT) learning frameworks. We find that while LLM informatics techniques can come close to traditional methods, they generally underperform in terms of predictive accuracy and computational efficiency. The fine-tuned Llama-3 model consistently outperforms GPT-3.5, likely due to the flexibility and tunability of the open-source architecture. Additionally, ST learning proves more effective than MT for LLMs, which struggle to exploit cross-property correlations-a significant and known advantage of traditional methods. The analysis of molecular embeddings learned by the models provides insight into the inner workings of the LLMs, revealing fundamental limitations of general-purpose LLMs in capturing nuanced chemo-structural information compared to the handcrafted features and domain-specific embeddings utilized in the traditional methods. These findings offer insights into the interplay between molecular embeddings and natural language processing, and provide guidance for LLM model selection within the context of polymer informatics.