LTVAE: Hybrid LSTM-Transformer VAE Architecture for Molecular SMILES Representation

A hybrid LSTM-Transformer Variational Autoencoder (LTVAE) framework is introduced in this paper for learning compact and chemically meaningful latent representations from SMILES strings, providing a SMILES-to-latent-to-SMILES representation model trained on ZINC15 and evaluated on ZINC15 and an out-of-distribution dye dataset. In the proposed model, sequential and structural dependencies in molecular strings are encoded using a bidirectional LSTM, while a Transformer-based decoder is employed to reconstruct valid molecular structures in the evaluated settings. When trained on a curated ZINC15 dataset, the framework achieves reasonable reconstruction performance, with 98.58% token-level accuracy, 92.19% string-level accuracy, a 99.30% chemical validity ratio, and an average Levenshtein distance of 0.29 using a 64-dimensional latent space. Reconstruction under an out-of-distribution evaluation setting is evaluated on a structurally distinct dye dataset, where 95.56% token-level accuracy and 85.30% validity are achieved. The learned latent space preserves trends aligned with both general molecular descriptors and dye-relevant structural descriptors, particularly conjugation length and aromatic surface fraction, with Fsp ${}^3$ providing complementary information on molecular saturation. Controlled perturbations in the latent space suggest smooth and chemically consistent structural variations, illustrating the continuity and interpretability of the learned latent manifold.