Detecting COVID-19 vaccine hesitancy in India: a multimodal transformer based approach

COVID-19 has emerged as the greatest threat in recent times, causing extensive mortality and morbidity in the entire world. India is among the highly affected countries suffering severe disruptions due this pandemic. To overcome the adverse effects of COVID-19, vaccination has been identified as the most effective preventive measure globally. However, a growing amount of hesitancy has been observed among the general public regarding the efficacy and possible side-effects of vaccination. Such hesitancy may proved to be the greatest hindrance towards combating this deadly pandemic. This paper introduces a multimodal deep learning method for Indian Twitter user classification, leveraging both content-based and network-based features. To explore the fundamental features of different modalities, improvisations of transformer models, BERT and GraphBERT are utilized to encode the textual and network structure information. The proposed approach thus integrates multiple data representations, utilizing the advances in both transformer based deep learning as well as multimodal learning. Experimental results demonstrates the efficacy of proposed approach over state of the art approaches. Aggregated feature representations from multiple modalities embed additional information that improves the classification results. The findings of the proposed model has been further utilized to perform a study on the dynamics of COVID-19 vaccine hesitancy in India.