Predicting Head Movement in Panoramic Video: A Deep Reinforcement Learning Approach.

Area of Science:

• Computer Vision
• Human-Computer Interaction
• Machine Learning

Background:

• Panoramic video offers immersive experiences by allowing users to control their field of view (FoV) via head movement (HM).
• Understanding human attention patterns, reflected in HM, is crucial for modeling user interaction with panoramic content.

Purpose of the Study:

• To establish a database of human head movements in panoramic video sequences.
• To investigate the predictability of human head movements using deep reinforcement learning (DRL).
• To propose and validate a DRL-based approach for predicting head movement positions in panoramic videos.

Main Methods:

• Collection of a comprehensive database detailing subjects' head movements (HM) during panoramic video viewing.
• Application of deep reinforcement learning (DRL) to learn and predict human HM scanpaths.
• Development of two DRL-based HM prediction (DHP) versions: offline-DHP for generating HM maps and online-DHP for real-time prediction.

Main Results:

• High consistency observed in human HM data across different subjects.
• DRL effectively predicts HM positions by imitating human scanpaths.
• Experimental validation confirms the effectiveness of both offline-DHP and online-DHP in predicting HM positions.

Conclusions:

• Human head movements in panoramic videos exhibit predictable patterns.
• The proposed DRL-based HM prediction (DHP) approach accurately forecasts viewer attention.
• The learned offline-DHP model significantly enhances the performance of the online-DHP system for real-time applications.