The value-complexity trade-off for reinforcement learning based brain-computer interfaces.

Area of Science:

• Neuroscience
• Computer Science
• Machine Learning

Background:

• Reinforcement learning (RL) is emerging in brain-computer interfaces (BCI), utilizing neural error signals for reward feedback.
• The reliability of RL-BCI hinges on accurately decoding noisy neural error signals.
• Existing methods require principled approaches to manage noise in RL-BCI.

Purpose of the Study:

• To develop a principled method for optimizing policy complexity in RL problems with noisy rewards, specifically for RL-BCI.
• To utilize the info-RL (IRL) algorithm to characterize maximal obtainable value under varying noise levels.
• To extract optimal robust policies for RL-BCI under different noise conditions.

Main Methods:

• Employed the info-RL (IRL) algorithm, which balances expected value against informational cost to yield optimal low-complexity policies.
• Characterized the maximal obtainable value under different noise levels using IRL.
• Simulated scenarios with Gaussian noise to analyze policy complexity and its dependence on reward magnitude and variance.

Main Results:

• Optimal policy complexity depends on reward magnitude, not reward variance, in Gaussian noise settings.
• Reward variance influences the favorability of lower complexity solutions.
• Demonstrated the application of the analysis for selecting optimal robust policies for RL-BCI using EEG data.

Conclusions:

• The proposed framework provides a principled method for determining optimal policy complexity in noisy RL reward scenarios.
• This approach is particularly beneficial for RL-BCI, potentially minimizing training time.
• The method enables more dynamic and robust shared control in RL-BCI systems across various conditions.