Domain-generalized representation learning for cross-chemical-family toxicity prediction

Traditional QSAR toxicity models are, in general, assessed with random train-test splits where structural overlaps between train and test compounds are allowed. This usually results in an inflated predictive performance. Therefore, this paper looks at the problem of toxicity prediction under the structural distribution shift and checks if representation-level invariance can contribute to better cross-family generalization. A set of 1792 structurally diverse organic molecules for which toxicity data (e.g., Tetrahymena pyriformis pIGC₅₀) were determined experimentally was modeled with physicochemical descriptors. In order to depict the realistic scenarios of model use, the leave-one-cluster-out (LOCO) protocol was applied to enforce strict structural separation of training and test domains. Baseline neural models lost a lot of their prediction accuracy under LOCO versus random splits, thus exposing a very large generalization gap. On the other hand, invariant learning methods such as invariant risk minimization, contrastive alignment, and domain-adversarial training managed not only to reduce the cross-domain error but also to make residual distributions more stable. The embedding of latent space further demonstrated that invariance helps to get rid of cluster-specific signals while keeping toxicity-relevant gradients intact. From a practical perspective, these results suggest that robust in silico predictive toxicology, further under structural distribution shift, can be achieved through domain-aware validation and invariant representation learning.