Learning with sparse reward in a gap junction network inspired by the insect mushroom body

Area of Science:

• Neuroscience
• Machine Learning
• Computational Biology

Background:

• Animals face challenges learning with sparse rewards, where reinforcement is delayed until task completion.
• Conventional reinforcement learning algorithms struggle with this 'distal' or 'sparse' reward problem.
• Insect mushroom bodies exhibit parallel fibers with potential axo-axonal gap junction connections forming resistive networks.

Purpose of the Study:

• To investigate a novel algorithm for reinforcement learning in scenarios with sparse rewards.
• To explore the potential of axo-axonal gap junctions in neural circuits as a model for decision-making.
• To propose a graph-based approach for encoding task structure through adaptive connections.

Main Methods:

• Developed a reinforcement learning algorithm inspired by resistive networks formed by gap junctions.
• Modeled task states as active nodes and state transitions as connections with adaptive weights.
• Simulated current flow to a target state to guide decision-making in a graph structure.

Main Results:

• Demonstrated efficient reinforcement learning in tasks with sparse rewards using the proposed algorithm.
• Showcased how adaptive gap junction weights can modulate connections to encode task structure.
• The algorithm effectively guides decision-making through a graph representation of the task.

Conclusions:

• The proposed algorithm offers an efficient solution for reinforcement learning under sparse rewards.
• The model provides a plausible computational account for the function of insect mushroom bodies.
• Axo-axonal gap junctions may play a crucial role in learning and decision-making in biological systems.