HMM for discovering decision-making dynamics using reinforcement learning experiments.

Area of Science:

• Computational psychiatry
• Neuroscience
• Behavioral economics

Background:

• Major depressive disorder (MDD) is a leading cause of disability, with diagnosis and treatment complicated by its heterogeneity.
• Abnormalities in reward processing are increasingly recognized as potential behavioral markers for MDD.
• Traditional reinforcement learning (RL) models may not fully capture the complexity of decision-making in MDD, suggesting strategy switching.

Purpose of the Study:

• To investigate how decision-making strategy dynamics influence reward learning in individuals with MDD.
• To propose and validate a novel computational framework for analyzing reward-based decision-making in MDD.

Main Methods:

• Developed a novel reinforcement learning-hidden Markov model (RL-HMM) framework to analyze reward-based decision-making.
• The RL-HMM accommodates strategy switching between RL-based choices and random choices, with a continuous RL state space and time-varying transitions.
• Employed an efficient Expectation-maximization (EM) algorithm for parameter estimation and nonparametric bootstrap for statistical inference.

Main Results:

• The RL-HMM framework demonstrated robust performance in extensive simulation studies.
• Application to the Establishing Moderators and Biosignatures of Antidepressant Response in Clinical Care (EMBARC) study revealed reduced RL engagement in MDD patients compared to healthy controls.
• Lower RL engagement in MDD was associated with brain activity in negative affect circuitry during an emotional conflict task.

Conclusions:

• The proposed RL-HMM framework effectively models strategy switching in reward-based decision-making.
• MDD is characterized by altered reward learning dynamics, specifically reduced engagement in reinforcement learning.
• These findings link reward processing abnormalities in MDD to neural activity in affective circuits.