Evaluating In-Context Learning in Large Language Models for Molecular Property Regression.

Area of Science:

• Artificial Intelligence
• Computational Chemistry
• Machine Learning

Background:

• Large language models (LLMs) excel at natural language tasks.
• Their capability for in-context learning (ICL) in scientific regression is not well understood.
• Assessing LLM performance in scientific domains requires specialized evaluation frameworks.

Purpose of the Study:

• To systematically evaluate the in-context learning abilities of seven large language models (LLMs) in scientific regression tasks.
• To investigate LLM performance on molecular property prediction under controlled conditions designed to isolate shortcut learning and induce out-of-distribution (OOD) behavior.
• To compare LLM performance against traditional machine learning (ML) baselines.

Main Methods:

• A controlled framework of 56 transformed tasks was used to assess seven LLMs on molecular property prediction.
• Tasks were designed to isolate shortcut learning and induce functional out-of-distribution (OOD) behavior.
• Performance was evaluated by comparing LLM results with machine learning (ML) baselines.

Main Results:

• LLMs achieved near-perfect performance on raw molecular weight prediction, likely due to shortcut cues.
• LLM performance significantly deteriorated under nonlinear transformations of the data.
• Machine learning (ML) baselines demonstrated greater robustness, leading to a performance crossover where ML outperformed LLMs.
• Meta-analysis identified distributional descriptors and structure-activity landscape indices (SALI) as predictors of task favorability.

Conclusions:

• LLMs' in-context learning for scientific regression is limited and susceptible to shortcut learning.
• Machine learning models offer more robust and reliable performance for molecular property prediction, particularly in out-of-distribution scenarios.
• Distributional descriptors and SALI can guide the selection of appropriate AI/ML approaches for chemistry applications.