State-space Model Based Inverse Reinforcement Learning for Reward Function Estimation in Brain-machine Interfaces.

Area of Science:

• Neuroscience
• Machine Learning
• Biomedical Engineering

Background:

• Reinforcement learning (RL) is a promising approach for brain-machine interfaces (BMIs) but relies on effective reward signals for decoder training.
• Designing accurate reward signals is challenging, particularly for complex tasks involving high-dimensional neural data.
• Inverse reinforcement learning (IRL) estimates reward functions from neural activity, but struggles with the large state-action spaces of multi-channel neural data.

Purpose of the Study:

• To propose a novel state-space model based inverse Q-learning (SSM-IQL) method to enhance IRL performance in BMIs.
• To address the challenge of high-dimensional neural activity in IRL by extracting hidden brain states.
• To improve the accuracy and stability of internal reward function estimation for RL-based BMIs.

Main Methods:

• Developed a state-space model (SSM) to extract latent brain states from high-dimensional neural recordings.
• Integrated the SSM with inverse Q-learning (IQL) to create the SSM-IQL algorithm.
• Validated the SSM-IQL method using real neural data from rats performing a two-lever discrimination task.

Main Results:

• The proposed SSM-IQL method demonstrated more accurate and stable estimation of the internal reward function compared to the traditional IQL algorithm.
• The state-space model effectively reduced the dimensionality of neural data by extracting relevant hidden brain states.
• Preliminary results indicate significant improvements in reward function estimation for RL-based BMI applications.

Conclusions:

• The SSM-IQL method offers a robust solution for estimating reward functions in complex BMI tasks.
• Integrating state-space models with IRL is a viable strategy for improving neural decoding in BMIs.
• This approach holds potential for advancing the design and performance of future RL-based brain-machine interfaces.