Identification of a Histone Deacetylase 8 Inhibitor through Drug Screenings Based on Machine Learning

Histone deacetylase 8 (HDAC8) is a zinc-dependent HDAC that catalyzes the deacetylation of nonhistone proteins. It is involved in cancer development and HDAC8 inhibitors are promising candidates as anticancer agents. However, most reported HDAC8 inhibitors contain a hydroxamic acid moiety, which often causes mutagenicity. Therefore, we used machine learning for drug screening and attempted to identify non-hydroxamic acids as HDAC8 inhibitors. In this study, we established a prediction model based on the random forest (RF) algorithm for screening HDAC8 inhibitors because it exhibited the best predictive accuracy in the training dataset, including data generated by the synthetic minority over-sampling technique (SMOTE). Using the trained RF-SMOTE model, we screened the Osaka University library for compounds and selected 50 virtual hits. However, the 50 hits in the first screening did not show HDAC8-inhibitory activity. In the second screening, using the RF-SMOTE model, which was established by retraining the dataset including 50 inactive compounds, we identified non-hydroxamic acid 12 as an HDAC8 inhibitor with an IC₅₀ of 842 nM. Interestingly, its IC₅₀ values for HDAC1 and HDAC3-inhibitory activity were 38 and 12 µM, respectively, showing that compound 12 has high HDAC8 selectivity. Using machine learning, we expanded the chemical space for HDAC8 inhibitors and identified non-hydroxamic acid 12 as a novel HDAC8 selective inhibitor.