WinDB: HMD-free and Distortion-free Panoptic Video Fixation Learning

Area of Science:

• Computer Vision
• Human-Computer Interaction
• Machine Learning

Background:

• Current fixation collection for panoptic video relies on head-mounted displays (HMDs).
• HMD-based methods suffer from "blind zooms," leading to incomplete fixation data.
• This limits the training of deep models for predicting salient regions in complex scenes.

Purpose of the Study:

• To introduce a novel fixation collection method for panoptic video that overcomes HMD limitations.
• To release a new dataset for training and evaluating fixation prediction models.
• To develop a network capable of handling fixation shifting phenomena.

Main Methods:

• Developed the auxiliary window with dynamic blurring (WinDB) approach for HMD-free fixation collection.
• Created the PanopticVideo-300 dataset with 300 panoptic clips and 225 categories.
• Proposed the fixation shifting network (FishNet) to address fixation shifting.

Main Results:

• WinDB effectively collects fixation data without "blind zooms," reflecting regional importance.
• The PanopticVideo-300 dataset captures frequent "fixation shifting," a previously overlooked phenomenon.
• FishNet demonstrates effectiveness in handling fixation shifting for improved prediction.

Conclusions:

• The WinDB approach and PanopticVideo-300 dataset offer a significant advancement in fixation data collection.
• The developed FishNet model addresses the challenges of fixation shifting in panoptic video.
• These contributions pave the way for new research and applications in 360° environments.