Comprehensive evaluation of molecule property prediction with ChatGPT.

Area of Science:

• Artificial Intelligence
• Computational Chemistry
• Drug Discovery

Background:

• Large Language Models (LLMs) like ChatGPT exhibit broad task versatility.
• Their application in specialized scientific domains, such as drug discovery, requires thorough evaluation.

Purpose of the Study:

• To comprehensively assess ChatGPT's capabilities in predicting molecular properties for drug discovery.
• To investigate the impact of prompt engineering and data sampling on prediction accuracy.
• To analyze the potential and limitations of ChatGPT compared to existing specialized models.

Main Methods:

• Evaluated ChatGPT on 53 ADMET-related endpoints for molecule property prediction.
• Investigated the effects of various prompt settings, including few-shot learning with scaffold sampling.
• Compared ChatGPT's performance against established, task-specific models.

Main Results:

• ChatGPT achieved satisfactory prediction outcomes, competitive with specialized models, under optimized prompt conditions.
• Prompt settings, particularly few-shot example selection and scaffold sampling, significantly influenced performance.
• Prediction accuracy was notably dependent on the quality of provided examples.

Conclusions:

• ChatGPT demonstrates significant potential for molecule property prediction in drug discovery with appropriate prompt strategies.
• The reliance on example quality presents a limitation for widespread practical application.
• This study provides insights for future LLM development and evaluation in scientific fields.