Structured matching models in multimodal information fusion: An optimized Kuhn-Munkres algorithm.

Area of Science:

• Human-Computer Interaction
• Multimodal Interaction Design
• Artificial Intelligence

Background:

• Integrating diverse data streams (speech, vision, text) in multimodal systems is challenging due to structural, temporal, and volume differences.
• These mismatches lead to inefficiency and poor user experiences in current multimodal interaction designs.
• Existing methods struggle with effective and accurate fusion of information from varied modalities.

Purpose of the Study:

• To enhance both the efficiency and accuracy of multimodal information fusion.
• To develop an optimized strategy for integrating diverse data streams in intelligent interaction systems.
• To improve user satisfaction and computational performance in multimodal applications.

Main Methods:

• Utilized publicly available datasets (CMU-MOSI, IEMOCAP) for speech, visual, and textual data.
• Applied preprocessing techniques including noise reduction, feature extraction (MFCCs, keypoint detection), and temporal alignment.
• Proposed an improved Kuhn-Munkres algorithm with dynamic weighting and a cross-modal correlation matrix for robust multimodal matching.

Main Results:

• Achieved a 28.2% improvement in multimodal information matching accuracy compared to baseline methods.
• Increased integration efficiency by 18.7% and reduced average computation time by 15.4%.
• Reported a 19.5% increase in user satisfaction ratings, validated by satisfaction surveys.

Conclusions:

• The enhanced Kuhn-Munkres algorithm offers a novel and effective optimization strategy for multimodal information integration.
• Dynamic weighting and correlation matrix constraints are critical for improving matching robustness and efficiency.
• The study provides substantial theoretical value and broad applicability for next-generation intelligent interaction and human-computer collaboration systems.