Transformer Learning in Sequence-Based Drug Design Depends on Compound Memorization and Similarity of...

Area of Science:

• Computational chemistry
• Artificial intelligence in drug discovery

Background:

• Chemical language models (CLMs), especially transformers, are used for generative molecular design.
• Their black-box nature hinders the interpretation of predictions at the molecular level.

Purpose of the Study:

• To investigate the learning characteristics of transformer CLMs in sequence-based compound design.
• To understand how these models learn and predict molecular properties.

Main Methods:

• Utilized sequence-based compound design as a model system.
• Performed systematic control calculations by modifying protein sequences and sequence-compound pairs.
• Analyzed compound reproducibility based on data similarity and memorization.

Main Results:

• Compound reproducibility was primarily driven by similarity between training and test data and memorization.
• Transformer CLMs did not learn specific chemical or biological sequence information.
• Model predictions are based on memorization and statistical correlations, not genuine chemical understanding.

Conclusions:

• Transformer CLMs in molecular design operate largely through memorization rather than learning specific chemical rules.
• Over-interpretation of CLM outputs can be avoided by understanding their reliance on statistical correlations.
• This knowledge guides the appropriate application of transformer-based CLMs in molecular design.