5-HT 2A and 5-HT 2C receptor antagonism differentially modulate reinforcement learning and cognitive flexibility:...

Cognitive flexibility, the ability to adapt behaviour in response to a changing environment, is disrupted in several neuropsychiatric disorders, including obsessive-compulsive disorder and major depressive disorder. Evidence suggests that flexibility, which can be operationalised using reversal learning tasks, is modulated by serotonergic transmission. However, how exactly flexible behaviour and associated reinforcement learning (RL) processes are modulated by 5-HT action on specific receptors is unknown.

We investigated the effects of 5-HT_2A receptor (5-HT_2AR) and 5-HT_2C receptor (5-HT_2CR) antagonism on flexibility and underlying RL mechanisms.

Thirty-six male Lister hooded rats were trained on a touchscreen visual discrimination and reversal task. We evaluated the effects of systemic treatments with the 5-HT_2AR and 5-HT_2CR antagonists M100907 and SB-242084, respectively, on reversal learning and performance on probe trials where correct and incorrect stimuli were presented with a third, probabilistically rewarded, stimulus. Computational models were fitted to task choice data to extract RL parameters, including a novel model designed specifically for this task.

5-HT_2AR antagonism impaired reversal learning only after an initial perseverative phase, during a period of random choice and then new learning. 5-HT_2CR antagonism, on the other hand, impaired learning from positive feedback. RL models further differentiated these effects. 5-HT_2AR antagonism decreased punishment learning rate (i.e. negative feedback) at high and low doses. The low dose also decreased reinforcement sensitivity (beta) and increased stimulus and side stickiness (i.e., the tendency to repeat a choice regardless of outcome). 5-HT_2CR antagonism also decreased beta, but reduced side stickiness.

These data indicate that 5-HT_2A and 5-HT_2CRs both modulate different aspects of flexibility, with 5-HT_2ARs modulating learning from negative feedback as measured using RL parameters and 5-HT_2CRs for learning from positive feedback assessed through conventional measures.