Can LLMs Solve Solubility Tasks? The SoluBench Benchmark for Pure and Mixed Solvent Systems

Area of Science:

• Computational Chemistry
• Drug Discovery
• Artificial Intelligence

Background:

• Solubility prediction is crucial in chemistry and drug discovery.
• The application of large language models (LLMs) to solubility prediction is an emerging area.
• Existing benchmarks for evaluating LLMs in this domain are limited.

Purpose of the Study:

• To introduce SoluBench, a novel benchmark for assessing LLM performance on solubility prediction tasks.
• To evaluate a diverse range of LLMs on their ability to handle solubility problems in pure and mixed solvent systems.
• To identify current limitations and capabilities of LLMs in chemical solubility assessment.

Main Methods:

• Development of SoluBench, a benchmark with 9806 questions across four complexity levels (pairwise solvent comparison, single-best solvent selection, cosolvent effect prediction, pairwise compound comparison).
• Utilizing experimental data from BigSolDB 2.0 and MixtureSolDB datasets to ground the benchmark.
• Systematic evaluation of over 20 proprietary and open-source LLMs, including Gemini 3 Flash.

Main Results:

• Frontier proprietary LLMs demonstrate strong performance on solvent-oriented tasks (Tasks 1-3), with Gemini 3 Flash achieving high accuracy.
• The solute-focused Task 4 presents significant challenges, particularly for polar aprotic solvents, requiring explicit reasoning.
• A universal performance bottleneck was observed for large solutes (Molecular Weight > 500 Da) across all tasks.

Conclusions:

• LLMs possess intrinsic capabilities for qualitative solubility assessment relevant to practical applications.
• Significant limitations persist in LLMs' understanding of molecular structure, impacting performance on complex solubility prediction tasks.
• Further development is needed to enhance LLMs' molecular structure comprehension for advanced chemical analysis.