UBG: An Unreal BattleGround Benchmark With Object-Aware Hierarchical Proximal Policy Optimization

Area of Science:

• Artificial Intelligence
• Robotics
• Computer Vision

Background:

• Deep reinforcement learning (DRL) shows promise in simulations but struggles with real-world application due to environmental limitations.
• Existing simulation environments lack the visual fidelity, complexity, and task diversity needed for advanced DRL research.
• Bridging the gap between simulated and real-world DRL performance requires more sophisticated and realistic training grounds.

Purpose of the Study:

• To develop a novel, realistic 3D simulation environment for advancing deep reinforcement learning.
• To introduce a new DRL algorithm designed to handle complex, open-world scenarios.
• To evaluate the effectiveness of the proposed environment and algorithm in enabling human-level decision-making for DRL agents.

Main Methods:

• Development of Unreal BattleGround (UBG), a 3D open-world first-person shooter game using the Unreal Engine, featuring variable complexity, random scenes, and diverse tasks.
• Proposal of the object-aware hierarchically proximal policy optimization (OaH-PPO) method, incorporating a two-level hierarchy for option control and subtask mastery.
• Integration of an object-aware module for depth detection, potential-based intrinsic reward shaping for exploration, and annealing imitation learning for guided initialization.

Main Results:

• The UBG environment demonstrates broad applicability for training and testing DRL agents in complex scenarios.
• The OaH-PPO method significantly enhances DRL agent performance within the UBG benchmark.
• Experimental results validate the effectiveness of the proposed hierarchical approach and its supporting modules.

Conclusions:

• Unreal BattleGround (UBG) provides a robust and realistic platform for overcoming DRL limitations in complex environments.
• The object-aware hierarchically proximal policy optimization (OaH-PPO) method offers an effective solution for improving DRL agent capabilities.
• The developed UBG environment and OaH-PPO algorithm represent a significant step towards deploying DRL in real-world applications.