Attention-Guided Neural Networks for Full-Reference and No-Reference Audio-Visual Quality Assessment

Area of Science:

• Computer Science
• Signal Processing
• Multimedia Systems

Background:

• Mobile internet usage drives demand for audio-visual (A/V) content.
• A/V compression for transmission causes signal distortions, degrading user Quality of Experience (QoE).
• Existing objective audio-visual quality assessment (AVQA) methods often neglect the combined impact of audio and visual information.

Purpose of the Study:

• To develop an objective AVQA architecture for multi-modal signals.
• To improve the accuracy of predicting end-user QoE by considering both audio and visual components.
• To create a flexible AVQA model applicable to both full-reference and no-reference scenarios.

Main Methods:

• Proposed a novel AVQA architecture utilizing attentional neural networks.
• Employed an attention prediction model to identify salient video regions.
• Used pre-trained Convolutional Neural Networks (CNNs) for feature extraction and Gated Recurrent Unit (GRU) networks for temporal modeling of A/V signals.

Main Results:

• The proposed model demonstrated superior performance compared to state-of-the-art AVQA methods on benchmark datasets (LIVE-SJTU and UnB-AVC).
• The architecture effectively fuses audio and visual features to predict a final quality score.
• The model's flexibility allows application in both full-reference and no-reference AVQA tasks.

Conclusions:

• The developed attentional neural network-based AVQA model offers a significant advancement in objective quality assessment.
• Considering multi-modal A/V signals is crucial for accurate QoE prediction.
• Public availability of the code will foster further research and development in AVQA.