Molecular basis defining the selectivity of substituted isoquinolinones for the melatonin MT₂ receptor

Melatonin MT₁ and MT₂ receptors represent attractive drug targets for the treatment of various disorders. However, the high conservation of the melatonin binding pocket has hindered the development of subtype-selective compounds. By leveraging on the recently resolved crystal structures of MT₁ and MT₂ receptors, this study aims to elucidate the structural basis of MT₂-selectivity of a panel of isoquinolinone derivatives. Molecular modelling and ligand docking approaches were employed to predict residues involved in forming interactions with the MT₂-selective isoquinolinones. Seven conserved residues (Asn¹⁷⁵, His²⁰⁸, Trp²⁶⁴, Asn²⁶⁸, Gly²⁷¹, Tyr²⁹⁴ and Tyr²⁹⁸) were selected as targets for site-directed mutagenesis. Ca²⁺ mobilization, cAMP inhibition, phosphorylation of extracellular signal-regulated kinase, and ligand binding assays were performed to functionally characterize the receptor mutants in transfected CHO cells. Unlike melatonin, isoquinolinones bearing a 3-methoxybenzyloxyl substituent were unaffected by alanine substitution at His²⁰⁸ of MT₂. Although alanine substitutions at Tyr²⁹⁴ or Tyr²⁹⁸ reduced the potency of melatonin and some isoquinolinones on MT₂, similar mutations on MT₁ allowed five hitherto ineffective isoquinolinones to act as agonists. An isoquinolinone antagonist bearing a 4-methoxybenzyloxyl moiety turned into an agonist at MT₂ mutants with alanine substitutions at His²⁰⁸, Tyr²⁹⁴ or Tyr²⁹⁸. A subset of residues is apparently involved in forming a hydrophobic binding cavity to confer selectivity upon the aromatic substituent of isoquinolinone compounds. Two conserved tyrosine residues on transmembrane helix 7 may confer ligand selectivity at MT₁ and MT₂ receptors, while a conserved histidine on transmembrane helix 5 is apparently involved in receptor activation.