Serotonin 6 receptor blockade reduces repetitive behavior in the BTBR mouse model of autism spectrum disorder

Autism spectrum disorder (ASD) is characterized by the expression of restricted repetitive behaviors (RRBs) and impairments in social recognition and communication. Previous studies have found that specific serotonin (5-HT) receptor modulation can attenuate repetitive behaviors expressed in specific mouse strains. The present study examined how 5-HT6 receptor blockade impacts the expression of repetitive behaviors in two different mouse strains that demonstrate elevated restricted, repetitive behavior and impairments in social behavior. BTBR T+ Itpr3tf /J (BTBR), C58/J (C58) and control C57BL/6J strains were behaviorally tested after acute treatment with the 5-HT6 receptor antagonist BGC 20-761 (BGC) or vehicle. BTBR mice express high levels of self-grooming behavior while C58 mice display high rates of repetitive jumping behavior. Similarly, the effect of 5-HT6 receptor blockade was also tested on social approach behaviors in both strains. BGC significantly reduced repetitive grooming in both female and male BTBR mice compared to vehicle-treated BTBR mice. BGC treatment did not attenuate social approach impairments in either female or male BTBR mice compared to vehicle-treated BTBR mice. Follow-up dose response studies were conducted on repetitive grooming and locomotor activity in BTBR mice. All doses reduced repetitive grooming in female and male BTBR mice. Acute treatment with BGC only reduced locomotor activity with the lower doses. In C58 mice, BGC treatment did not significantly attenuate flipping or general social approach behaviors. Instead, BGC significantly increased social sniff time in female C58 mice. While 5-HT6 receptor blockade did not attenuate the social impairments found in BTBR mice, this treatment did increase sniff time in female C58 mice. Although the lower doses of BGC deduced locomotion, the higher dose attenuated repetitive grooming in BTBR mice while sparing locomotor activity. Together these findings suggest the therapeutic effects of 5-HT6 receptor blockade are complex and may be specific to the types of repetitive behaviors expressed.