XFlow: Cross-Modal Deep Neural Networks for Audiovisual Classification

Area of Science:

• Computer Science
• Artificial Intelligence
• Machine Learning

Background:

• Multimodal tasks require integrating information from diverse data sources for enhanced representation learning.
• Exploiting cross-modal correlations (e.g., audio-visual) is crucial but challenging due to differing data characteristics.

Purpose of the Study:

• To propose novel deep learning architectures that enable effective dataflow and representation exchange between modalities.
• To improve upon existing multimodal deep learning algorithms by introducing early cross-modality fusion and extended cross connections.

Main Methods:

• Developed two deep learning architectures featuring multimodal cross connections (XFlow) for inter-feature extractor dataflow.
• Implemented a novel method for cross-modality fusion before individual modality feature extraction.
• Extended existing cross connections to handle data streams with compatible and incompatible data types.

Main Results:

• The proposed XFlow models achieved superior performance compared to baselines across AVletters, CUAVE, and the new Digits dataset, with improvements up to 11.5%.
• Learned representations from cross connections demonstrated increased discrimination ability and compatibility with lip-reading tasks.
• Achieved state-of-the-art results on benchmark multimodal datasets.

Conclusions:

• The novel cross-modal architectures effectively leverage audio-visual correlations for improved multimodal task performance.
• The proposed methods offer a more interpretable and powerful approach to multimodal representation learning.
• The XFlow architecture and the Digits dataset contribute valuable resources to the multimodal AI research community.