Outracing champion Gran Turismo drivers with deep reinforcement learning

Area of Science:

• Artificial Intelligence
• Robotics
• Computational Neuroscience

Background:

• Real-time decision-making in physical systems interacting with humans presents significant challenges.
• Automobile racing exemplifies these challenges, requiring complex tactical maneuvers and precise vehicle control at physical limits.
• Racing simulations like Gran Turismo accurately model these complex, multi-agent dynamics.

Purpose of the Study:

• To train artificial intelligence (AI) agents capable of competing at the highest level in the Gran Turismo racing simulation.
• To develop an integrated control policy combining speed and tactical decision-making for AI racers.
• To create a reward function that promotes competitive yet sportsmanlike behavior within the simulation's rules.

Main Methods:

• Utilized state-of-the-art, model-free, deep reinforcement learning algorithms.
• Implemented mixed-scenario training to expose agents to diverse racing conditions.
• Developed a specialized reward function to balance performance with adherence to sportsmanship norms.

Main Results:

• Trained AI agents, named Gran Turismo Sophy, demonstrated the ability to compete with world-class e-sports drivers.
• The AI agents exhibited exceptional speed and sophisticated tactical maneuvers.
• Gran Turismo Sophy successfully won a head-to-head competition against four top human Gran Turismo players.

Conclusions:

• The study showcases the potential of deep reinforcement learning for controlling complex dynamical systems in human-interactive domains.
• AI agents can achieve championship-level performance in simulated environments requiring both speed and strategic interaction.
• Controlling AI in domains with imprecise human norms, like sportsmanship, presents both possibilities and challenges.