Multimodal Deep Learning for Participatory Public Art Assessment: Integrating Computer Vision and Behavioral...

In this work, a deep learning-based multimodal framework for evaluating public engagement with public art is introduced by combining computer vision and behavioral analysis. Existing multimodal models have primarily been applied in domains such as educational settings and behavior analysis, while their application to public art assessment remains limited by challenges in representing subjective human factors, complex behaviors, and interactions across modalities. To address these challenges, the proposed framework employs a Transformer-based architecture to integrate multimodal information and model cross-modal interactions. Image and video inputs representing artworks and participant behaviors, including movement, gaze, and interaction, are processed using 3D convolutional neural networks to extract spatiotemporal features, followed by modality-specific encoding and adaptive fusion using cross-attention. The framework generates outputs related to engagement, emotion recognition, and interaction assessment. For component-level evaluation before real-world public art deployment, the framework was evaluated on three complementary benchmark datasets: NTU RGB+D, Kinetics-700, and AVA (Aesthetic Visual Analysis), representing behavioral, activity recognition, and aesthetic perception tasks. In the classification task, the accuracy obtained was 92.0% and the F1-score of 0.909 for NTU RGB+D, while that of Kinetics-700 was 89.0% and the F1-score of 0.878. The AVA dataset was evaluated using aesthetic-score regression, yielding a Pearson correlation coefficient of 0.87. These findings provide component-level evidence of computational effectiveness but do not constitute end-to-end validation in real-world public art settings.