Deletion of the 5-HT3A receptor reduces behavioral persistence and enhances flexibility

The 5-HT3A receptor is the only ionotropic serotonin receptor and has been implicated in cognitive functions, yet its specific role remains unclear. To investigate the contribution of the 5-HT3A subtype, we trained wild-type (C57BL/6 J) and 5-HT3A receptor knockout (KO) (Htr3a-/-) mice across a series of operant conditioning tasks and compared their behavioral performance. Following nose-poke training, both groups underwent a rule-switching task, extinction tests under fixed ratio (FR) and variable ratio (VR) reinforcement schedules, and a progressive ratio (PR) task to assess persistence. We found that 5-HT3A receptor knockout mice exhibited reduced responding during the extinction suggesting diminished behavioral persistence. The PR task result, in which KO mice showed a trend toward reduced responding, also supports this interpretation. Notably, however, knockout mice acquired the new rule in the switching task significantly faster than wild-type controls, indicating enhanced cognitive flexibility. These findings suggest that the 5-HT3A receptor plays a role in regulating the balance between behavioral persistence and flexibility, normally biasing this balance toward persistence under normal physiological conditions. This mechanism may underlie the therapeutic effect of 5-HT3A receptor antagonists in treatment-resistant obsessive-compulsive disorder (OCD).