A Hybrid PAC Reinforcement Learning Algorithm for Human-Robot Interaction

Area of Science:

• Artificial Intelligence
• Machine Learning
• Robotics

Background:

• Reinforcement learning (RL) algorithms for Markov decision processes (MDPs) often specialize as either model-based or model-free.
• Existing methods present trade-offs between sample efficiency and performance.
• A need exists for hybrid approaches that leverage the strengths of both methodologies.

Purpose of the Study:

• To introduce a novel hybrid probably approximately correct (PAC) reinforcement learning algorithm.
• To combine the advantages of model-free Delayed Q-learning and model-based R-max algorithms.
• To analyze the theoretical properties and practical performance of the proposed algorithm.

Main Methods:

• Development of the Dyna-Delayed Q-learning (DDQ) algorithm, a hybrid PAC-RL approach.
• Conducting a PAC analysis to derive the sample complexity of DDQ.
• Performing numerical simulations to compare DDQ against established PAC model-free and model-based algorithms.
• Implementing DDQ in a real-world pediatric motor rehabilitation setting using infant-robot interaction.

Main Results:

• The DDQ algorithm integrates model-free and model-based RL techniques effectively.
• DDQ consistently outperforms its constituent algorithms (Delayed Q-learning, R-max) in most scenarios.
• The algorithm exhibits superior sample efficiency compared to existing state-of-the-art PAC RL methods.
• Successful experimental validation in a pediatric motor rehabilitation context demonstrates practical utility.

Conclusions:

• The DDQ algorithm represents a significant advancement in PAC reinforcement learning for MDPs.
• Hybrid approaches can effectively bridge the gap between model-based and model-free RL.
• DDQ shows promise for improving sample efficiency and performance in complex real-world applications, such as assistive robotics.