Deep learning for assay nuisance compound detection using a gated co-attention graph embedding model (CAGE-Fusion)

In drug discovery, nuisance compounds are chemicals that interfere with biochemical or cell-based assays, often generating misleading signals unrelated to true activity. These molecules operate through diverse, context-dependent mechanisms, ranging from colloidal aggregation to covalent reactivity. They are often referred to as Pan-Assay Interference Compounds (PAINS), signal interferers, or frequent hitters, depending on their mode of action, and they lead to costly false positives and resource misallocation. They are difficult to distinguish from genuine pharmacological activity using rigid substructure filters. Although deep learning has emerged as a powerful alternative, most existing architectures process molecular graphs or SMILES sequences in isolation. We introduce CAGE-Fusion (Co-Attention Graph Embedding Fusion), a multimodal framework that employs a gated co-attention mechanism to enforce bidirectional information exchange between graph-based and sequence-based encoders. This architecture enables the iterative refinement of atom and token embeddings, yielding a chemically coherent representation that captures cross-modal dependencies. CAGE-Fusion demonstrates competitive predictive performance across several MoleculeNet benchmarks and achieves a macro-averaged PR-AUC of 0.73 and ROC-AUC of 0.94 on a comprehensive nuisance compound classification task. Beyond prediction, the model visualizes attention-weighted regions that contribute to a classification decision.Scientific contributionWe present CAGE-Fusion, a gated co-attention architecture that enables bidirectional information exchange between graph-based encoders and SMILES representations to learn cross-modal molecular embeddings. We evaluate the framework on standardized molecular property benchmarks and apply it in a focused case study on assay nuisance compound detection across four nuisance categories: aggregation, luciferase inhibition, reactivity, and promiscuity.