Comparing massively-multitask regression algorithms for drug discovery

Massively-multitask regression models (MMRMs) have revolutionized activity prediction for drug discovery. MMRMs trained on millions of compounds and many thousands of assays can predict bioactivity with accuracy comparable to 4-concentration IC₅₀ experiments. This report compares six MMRMs: pQSAR, Alchemite, MT-DNN, MetaNN, Macau and IMC. Models were trained by experts in each method, on identical sets of 159 kinase and 4276 diverse ChEMBL assays, employing realistically novel training/test set splits. Results were compared both qualitatively and with statistical rigor. Our use-case is imputing full bioactivity profiles for the very sparse compound collections on which the models were trained. MMRMs performed much better than the single-task random forest regression (ST-RFR) model. Five MMRMs train all models simultaneously, so must leave out test-set measurements from all assays to avoid leakage (here 25% of data), whereas one method trains models one-at-a-time, so only holds out test data for that assay (< 1% of data). Thus, all algorithms were compared both using 75/25 splits, and when possible, 99 + / < 1 splits. Many MMRM evaluations achieved similar accuracy when tested on the same split. However, when evaluated on 75/25 splits, all MMRMs performed much worse than when evaluated on 99 + / < 1% splits. Thus, while many MMRMs produce comparable final production models (trained on all the data), models that require 75/25 splits greatly underestimate the accuracy of the final models. While outstanding for imputations, MMRMs proved little better than ST-RFR for compounds very unlike the training collection. Thus, MMRMs are best for hit-finding, off-target, promiscuity, MoA, polypharmacology or drug-repurposing within the training collection. Since accuracy is not a deciding factor, other pros and cons of each method are also described.