HDAC8 Assistant: a QSAR-guided molecular generation platform for de novo design and virtual screening of HDAC8...

Histone deacetylase 8 (HDAC8) inhibitors are considered promising therapeutic agents for the treatment of cancer and several other diseases. In this study, regression QSAR models for HDAC8 inhibitors were developed using various types of molecular descriptors and machine learning algorithms. The best-performing weighted consensus model achieved an external predictive performance of Q²_test = 0.62 for the complete test set and 0.71 for compounds within the applicability domain, covering 77% of the external data. Model reliability was confirmed by five-fold cross-validation and y-randomization tests. The developed QSAR model was integrated into the HDAC8 Assistant web application (https://hdac8-assistant.streamlit.app/), which includes a de novo molecular generation module based on recurrent neural networks (SMILES-RNN) and reinforcement learning (REINVENT 2). For targeted discovery of novel inhibitors, an original reward function was proposed that combines QSAR-predicted activity with applicability domain control for generated compounds. This approach enables simultaneous optimization of predicted inhibitory activity while limiting extrapolation beyond the model applicability domain. HDAC8 Assistant also implements assessment of drug-likeness according to the Muegge criteria, as well as PAINS, toxicophore, and Brenk structural-alert detection. Molecular docking was performed for the most promising compounds generated by HDAC8 Assistant to analyse their interactions with the HDAC8 active site.