Science of music-based citizen science: How seeing influences hearing

Area of Science:

• Music Information Retrieval
• Citizen Science
• Human-Computer Interaction

Background:

• Citizen science projects often rely on visual annotation tasks.
• Music structure annotation is crucial for applications like music information retrieval and therapy.
• The impact of audiovisual versus unimodal input on music annotation quality is not well understood.

Purpose of the Study:

• To investigate the effect of different sensory inputs (audio, visual, audiovisual) on the accuracy of music structure segmentation by citizen scientists.
• To compare the effectiveness of unimodal (visual or audio) versus cross-modal (audiovisual) inputs for music annotation.

Main Methods:

• Participants annotated music segmentation boundaries from Beethoven's 32 Variations in C minor under varying audiovisual conditions.
• Segmentation boundaries were visualized using boundary credence profiles.
• Annotation quality was assessed using unbalanced optimal transport distance and F-measure, comparing against a cross-modal gold standard.

Main Results:

• Visual-only input yielded annotations closer to the audiovisual gold standard than audio-only input.
• Visual input enhanced global segmentation clarity, indicated by larger boundary credence profile peaks.
• Audio input proved more effective for discerning subtle structural nuances and resolving discrepancies.

Conclusions:

• Audiovisual input serves as cognitive scaffolding, improving citizen science music annotation.
• Visual cues are beneficial for identifying large-scale musical structures.
• Audio input is essential for capturing complex structural details and fine-tuning annotations.