Mapping the melatonin receptor. 5. Melatonin agonists and antagonists derived from tetrahydrocyclopent[b]indoles, tetrahydrocarbazoles and hexahydrocyclohept[b]indoles

Tetrahydrocyclopent[b]indoles, tetrahydrocarbazoles, and hexahydrocyclohept[b]indoles have been prepared as melatonin analogues to investigate the nature of the binding site of the melatonin receptor. The affinity of analogues was compared in a radioligand binding assay using chicken brain membranes and agonist and antagonist potency measured in clonal Xenopus laevis melanophore cells. Comparison of the N-acyl-3-amino-6-methoxytetrahydrocarbazoles (2) with N-acyl-4-(aminomethyl)-6-methoxy-9-methyltetrahydrocarbazoles (9) showed that the latter have much higher binding affinities for the chicken brain receptor. Comparison of N-acyl-1-(aminomethyl)-7-methoxy-4-methyltetrahydrocyclopent[b]ind oles (10), 6-methoxytetrahydrocarbazoles (9), and N-acyl-10-(aminomethyl)-2-methoxy-5-methylhexahydrocyclohept[b]ind oles (11) showed that the tetrahydrocarbazoles had the highest binding affinity with the cyclohept[b]indoles and the cyclopent[b]indoles having rather lower affinities. All of these observations are in agreement with our postulated model of melatonin orientation at the binding pocket in which the 3-amidoethane side chain is in a conformation close to the 5-methoxyl group, as is shown in the X-ray crystallographic structure of 9m and in the energy-minimized computed structures. Separation of the enantiomers of members from each of these three systems was accomplished by chiral HPLC. It was found that in all cases the (-)-enantiomer had a higher binding affinity than the (+)-enantiomer. An X-ray crystallographic analysis of the two enantiomers of 9a showed that the (+)-enantiomer had the (R) absolute stereochemistry. Since the sign of the Cotton curves, determined from circular dichroism studies, was the same for all (+)-enantiomers, it is assumed that the absolute stereochemistry at these centers is identical. In the Xenopus melanophore assay, the tetrahydrocarbazoles 2 (R = H) were mainly weak antagonists, while those with R = OMe were agonists. The biological behavior of the tetrahydrocarbazoles 9 (R = H) depended on R1, some being agonists and some antagonists, whereas those with R = OMe were generally agonists. Variation of the R and R1 groups in compounds of type 9 produced both agonists and antagonists. The tetrahydrocylopentaindoles 10 had similar biological properties to the corresponding analogues of 9, but the hexahydrocycloheptaindoles 11 showed a much greater propensity to be antagonists. In all cases the (S)-enantiomers were found to be more potent agonists than the (R)-enantiomers.