When and Why Hyperbolic Discounting Matters for Reinforcement Learning Interventions

Area of Science:

• Artificial Intelligence
• Cognitive Science
• Reinforcement Learning

Background:

• Human decision-making often exhibits hyperbolic discounting of future rewards.
• Current Reinforcement Learning (RL) models predominantly use exponential discounting for human behavior, simplifying planning.
• This discrepancy poses challenges for AI agents aiming to effectively guide human behavior.

Purpose of the Study:

• To investigate the trade-offs between computational cost and performance benefits of modeling humans with hyperbolic discounting.
• To develop and evaluate an AI policy that modifies human discounting behavior to achieve distant goals.
• To determine if approximating hyperbolic discounting with an exponential factor is computationally feasible and effective.

Main Methods:

• Derived a fixed exponential discount factor to approximate hyperbolic discounting in human models.
• Proved theoretical guarantees for the approximation, ensuring no necessary AI interventions are missed.
• Compared the approximation against the mean hazard rate method for reducing unnecessary interventions (false positives).

Main Results:

• The derived exponential approximation guarantees that AI agents will not miss crucial interventions.
• The approximation results in fewer false positives compared to the mean hazard rate method.
• Experimental results show that exponential approximations outperform true hyperbolic models in online learning scenarios.

Conclusions:

• Approximating hyperbolic discounting with a fixed exponential factor is a viable strategy for AI agents.
• This approach enhances the effectiveness of AI interventions by improving human goal-directed behavior.
• The surprising finding that exponential approximations excel in online learning warrants further investigation into AI-human interaction dynamics.