Disease relevance and replicability of deep learning gene expression prediction

Recent deep learning (DL) models predict average gene expression levels from DNA sequences with high overall correlation to measured values. We examine these DL models through the lens of disease research. The seemingly high overall performance of DL models is largely due to capturing whether genes are "On" or "Off", and to a lesser extent, disease-relevant expression level changes for "On" genes. Indeed, the more bimodal the gene expression distribution, the better the reported performance. We track the extent of this issue across tissues, molecular systems, and cancers. Compounding the model evaluation issue, we find inconsistencies between the published code and reported performance, highlighting the importance of versioning and publishing performance evaluation code. These findings indicate that the high reported performance of popular DL models falls unexpectedly short in disease applications, and that the problem of personalized genomic prediction remains far from solved in a disease context.