Event-centric multi-modal fusion method for dense video captioning.

Area of Science:

• Computer Science
• Artificial Intelligence
• Multimedia Processing

Background:

• Dense video captioning models often overlook inter-event relationships, impacting description consistency.
• Current methods primarily use visual features, neglecting multimodal information for event localization and description.

Purpose of the Study:

• To develop an event-centric approach for dense video captioning that captures temporal and semantic relationships between events.
• To enhance multimodal information fusion for more consistent and accurate video descriptions.

Main Methods:

• Exploited visual-audio cues for generating event proposals and improving event representations.
• Developed an attention-gating mechanism for dynamic fusion and regulation of multimodal information.
• Proposed an event-centric multimodal fusion approach for dense video captioning (EMVC).

Main Results:

• The EMVC model demonstrated impressive performance on benchmark datasets (ActivityNet Caption, YouCook2).
• The approach effectively captured relationships between events and fused multimodal information.
• Achieved superior results compared to existing state-of-the-art methods in dense video captioning.

Conclusions:

• The proposed EMVC approach significantly improves dense video captioning by integrating inter-event relationships and multimodal fusion.
• Event-centric processing and attention-gating mechanisms are crucial for enhancing video description quality.
• The model offers a promising direction for more comprehensive and consistent automatic video description.