Enhanced multimodal transformer framework for sarcasm detection with particle swarm optimization based feature fusion

Intelligent Human-Computer Interaction (HCI) has a pivotal place in sarcasm detection, given that sarcastic utterances may have an intended meaning that is not in the literal meaning of the spoken words. Sarcasm is also difficult to spot correctly since it varies not only in text but also in voice and other visual indicators like facial expression and body language. Unimodal sarcasm detection methods are generally limited by contextual ambiguity, the insensitivity of contextual paralinguistic cues, and decreased performance in situations where one modality is noisy or incomplete. These problems may cause unpredictable results and inadequate functionality in the real conversational context. To overcome these shortcomings, for sarcasm detection, we suggest an approach with different modalities which uses multi-modal cues like audio, video as well as text to provide complementary information from various communication channels. In the case of textual representation, the RoBERTa architecture was used to elicit deep contextual and semantic features of dialog transcripts. The HuBERT architecture was used to model acoustic features and is effective at representing speech representations in terms of prosodic and tonal variation based on sarcastic speech. Simultaneously, the ViViT architecture was used to process visual data and identify both spatial and temporal patterns of facial expressions and gestures in videos. Modality-specific feature representations are then combined with a Particle Swarm Optimization (PSO) based fusion strategy which decides best-found modality weights by minimizing classification loss. The fused representation is then optimized and then through the two-layer fully connected classifier to classify sarcasm. The suggested framework was evaluated with the publicly available MUStARD dataset. The model displayed an accuracy of 94% and the recall, precision and weighted F1-score were all equal to 0.94. Comparative evaluation also compared the proposed PSO-Multimodal Fusion Model with the unimodal models and also with the traditional multimodal fusion models such as attention-based model, Graph-based model, Generative Approach, LLM/Modeling-based model, and contrastive learning approach, and showed that the proposed multimodal fusion framework yields the highest weighted F1-score among the evaluated approaches. Experimental outcomes prove that the multimodal fusion methodology based on optimization enhances the performance of sarcasm detection in contrast to the unimodal and traditional fusion techniques, which indicate its efficiency in practical use in conversational AI.