A performance-driven hybrid text-image classification model for multimodal data.

Area of Science:

• Artificial Intelligence
• Machine Learning
• Computer Vision
• Natural Language Processing

Background:

• Deep learning models often struggle with multi-type data, limiting their effectiveness in real-world applications.
• Categorization tasks involving both textual and visual information require sophisticated approaches to handle diverse data modalities.

Purpose of the Study:

• To introduce the Hybrid Text-Image Categorization (HTIC) model, a novel deep learning architecture for processing multi-type data.
• To evaluate the performance of the HTIC model against other classification methods across various datasets.

Main Methods:

• The HTIC model employs a complex deep learning architecture, integrating VGG16 for image classification and Roberta with optimized CNNs for text classification.
• Multi-modal feature extraction layers are utilized to ensure compatibility between image and text representations.
• The model leverages Roberta's capabilities for extracting information from textual embeddings and capturing cross-modal patterns.

Main Results:

• The HTIC model demonstrated superior performance compared to other classification algorithms across five diverse datasets.
• The model exhibits enhanced generalizability and robustness to input data variations.
• The study highlights the effectiveness of hybrid modeling in advancing classification accuracy and interpretability.

Conclusions:

• The HTIC model represents a significant advancement in multimodal data analysis, offering improved classification accuracy and reliability.
• Its robust performance and generalizability make it suitable for real-world applications, including the analysis of NFT datasets.