Image classification with symbolic hints using limited resources

Area of Science:

• Machine Learning
• Computer Vision
• Natural Language Processing

Background:

• Traditional machine learning classification benchmarks often simplify real-world data structures.
• Integrating diverse data types, such as image and text metadata, presents a significant challenge.

Purpose of the Study:

• To develop and evaluate a novel late fusion method for incorporating "hints" into machine learning classification tasks.
• To demonstrate the effectiveness of this approach in image classification scenarios using text metadata.

Main Methods:

• A late fusion scheme was proposed, leveraging a conditional independence assumption to combine information from pre-trained image classifiers and text models.
• Model calibration was identified as a critical factor for successful fusion.
• The late fusion approach was compared against a mid-level fusion strategy utilizing support vector machines.

Main Results:

• The proposed late fusion scheme successfully integrated text metadata hints into image classification tasks.
• Model calibration was demonstrated to be crucial for achieving high performance in the fused model.
• Late fusion performance was comparable to mid-level fusion but with significantly lower computational overhead.

Conclusions:

• Late fusion offers an efficient and effective method for incorporating auxiliary information, or "hints," into machine learning classification.
• The approach is particularly promising for real-world applications where data is multimodal and complex.
• Further research into calibration techniques can enhance the utility of fusion methods in machine learning.