Machine learning driven LD₅₀ prediction for cancer risk assessment using modern molecular language models

Accurate assessment of chemical toxicity is fundamental to cancer research, where early identification of hazardous compounds is critical for prioritizing carcinogenicity testing, therapeutic safety evaluation, and regulatory decision-making.

We developed ChemModernBERT, a ModernBERT-based molecular language model pretrained using a curriculum learning strategy on more than 1.8 million SMILES strings to generate chemically informed sequence representations. Using a curated dataset of 8,898 compounds, we systematically compared four molecular representation learning approaches ChemBERT, ChemProp (a directed message passing neural network), ensemble learning, and ChemModernBERT for predicting median lethal dose (LD₅₀) values. Model performance was evaluated using standardized internal and external test sets.

ChemModernBERT achieved the lowest mean absolute error (MAE) on both internal (0.390) and external (0.393) evaluations and the highest coefficient of determination (R² = 0.521 on the external test set), outperforming ChemBERT, ChemProp, and ensemble models under identical experimental conditions. The small generalization gap between internal and external evaluations indicates strong transferability across chemically diverse compounds.

These findings demonstrate that curriculum-pretrained transformer architectures provide a scalable and accurate framework for large-scale toxicity prediction. Such models can support computational pipelines for carcinogenicity assessment, dose selection, and early-stage chemical safety evaluation.