Multimodal Stereoscopic Movie Summarization Conforming to Narrative Characteristics

Area of Science:

• Computer Vision
• Multimedia Processing
• Artificial Intelligence

Background:

• The proliferation of video data necessitates effective summarization techniques.
• Existing video summarization methods often struggle with the unique challenges of stereoscopic 3D content.
• There is a need for algorithms that balance summary compactness, enjoyability, and content coverage for 3D films.

Purpose of the Study:

• To propose a novel multi-stage, multimodal summarization process for stereoscopic 3D theatrical films.
• To develop a method for extracting short, representative video skims that adhere to narrative characteristics.
• To address the specific challenges posed by stereoscopic 3D video data in summarization.

Main Methods:

• Introduced a low-level video frame description method (frame moments descriptor) capturing luminance, color, optical flow, and stereoscopic disparity.
• Employed key-frame extraction and multimodal shot pruning, modeled as a matrix subset selection problem solved with evolutionary computing.
• Incorporated post-processing steps to manage audio-visual narrative prominence and reduce stereoscopic video defects.

Main Results:

• Successfully extracted informative frame features encompassing global and local image statistics.
• Developed a multimodal shot pruning technique that leverages character prominence in both visual and audio streams.
• Implemented post-processing to mitigate disorienting editing effects and reduce visual fatigue from stereoscopic defects.

Conclusions:

• The proposed method effectively generates compact and narrative-driven video summaries for stereoscopic 3D films.
• The multimodal approach enhances summary quality by considering both visual and audio content.
• This research contributes a novel solution for the under-explored area of 3D movie summarization.