Jove
Visualize
Contact Us
JoVE
x logofacebook logolinkedin logoyoutube logo
ABOUT JoVE
OverviewLeadershipBlogJoVE Help Center
AUTHORS
Publishing ProcessEditorial BoardScope & PoliciesPeer ReviewFAQSubmit
LIBRARIANS
TestimonialsSubscriptionsAccessResourcesLibrary Advisory BoardFAQ
RESEARCH
JoVE JournalMethods CollectionsJoVE Encyclopedia of ExperimentsArchive
EDUCATION
JoVE CoreJoVE BusinessJoVE Science EducationJoVE Lab ManualFaculty Resource CenterFaculty Site
Terms & Conditions of Use
Privacy Policy
Policies

Related Experiment Videos

A deep learning framework for emotion recognition in music using multimodal data fusion.

Runhua Li1

  • 1Student Affairs Office, Fuzhou University of Foreign Languages and Trade, Fuzhou, 350202, China. zaalabnell@hotmail.com.

Scientific Reports
|June 10, 2026
PubMed
Summary

This study introduces a deep learning framework for music emotion recognition using multimodal data. The novel approach enhances accuracy by better modeling temporal, harmonic, and hierarchical structures in music.

Related Concept Videos

Labeling Emotion01:20

Labeling Emotion

Emotional labeling is a cognitive process that involves identifying and naming one's emotions, such as anger, fear, happiness, or sadness. It allows individuals to recognize and express their internal emotional states, a critical aspect of emotional regulation and communication. Labeling emotions requires more than mere recognition; it also involves drawing upon memory and contextual cues to understand the current situation and apply a corresponding emotional label. For instance, feeling...
Physiology of Emotion01:20

Physiology of Emotion

The physiology of emotions is a multifaceted process involving the autonomic nervous system, brain structures, hormones, and neurotransmitters. This intricate interplay dictates how emotions manifest in the body and influence behavior.
Autonomic Nervous System
The autonomic nervous system (ANS) plays a critical role in emotional responses by regulating involuntary physiological functions. It consists of two main components: the sympathetic and parasympathetic systems. The sympathetic system...

You might also read

Related Articles

Articles linked to this work by shared authors, journal, and citation graph.

Sort by
Same author

Renal CT algorithm for evaluating clear cell carcinoma in solid renal masses ≤ 4 cm: a retrospective chinese two-center validation study.

Abdominal radiology (New York)·2026
Same author

Smoking as a determinant of rheumatoid arthritis: integrated evidence from epidemiological patterns, genetic signals, and mechanistic insights.

Archives of medical science : AMS·2026
Same author

Modified langendorff-free method for isolating cardiomyocytes and non-cardiomyocytes from adult mice.

BMC biotechnology·2026
Same author

The diagnostic value of multiple schemes of MRI standard features combined with risk factors and ultrasound for Placenta accreta spectrum.

BMC pregnancy and childbirth·2026
Same author

Reducing defects in ruthenium oxide <i>via</i> crystal engineering towards a durable acid oxygen evolution reaction.

Chemical communications (Cambridge, England)·2026
Same author

Two birds with one stone: Targeting METTL3 ameliorates doxorubicin-induced endothelial premature senescence and atherosclerosis while potentiating its antitumor efficacy.

Free radical biology & medicine·2026

Area of Science:

  • Affective Computing
  • Machine Learning
  • Music Information Retrieval

Background:

  • Existing music emotion recognition models struggle with cross-genre generalization, long-term dependencies, and hierarchical emotional structures.
  • Deep learning approaches require robust feature extraction and effective fusion of multimodal data (audio, lyrics, metadata).

Purpose of the Study:

  • To propose a novel deep learning framework for multimodal music emotion recognition.
  • To address limitations in existing models, particularly in generalization, temporal dependency modeling, and hierarchical emotional structure capture.
  • To improve the accuracy and robustness of music emotion recognition systems.

Main Methods:

  • Developed a Harmonic Semantic Encoder (HSE) with dual pathways (CNNs and Transformer) for acoustic feature extraction and long-range dependency modeling.
Keywords:
Contrastive alignmentDeep learningHarmonic encoderHuman–media interactionMultimodal data fusion

Related Experiment Videos

  • Introduced a Contrastive Harmonic Alignment (CHA) strategy with hierarchical contrastive objectives for enhanced representation learning.
  • Implemented a modality-aware attention mechanism for robust multimodal fusion (audio, lyrics, metadata), including handling missing data.
  • Designed a harmonic-aware attention mechanism to focus on emotionally salient frequency bands.

    • Main Results:

    • The proposed framework significantly outperformed strong baselines (SVM, CNN, CRNN, ResNet) on PMEmo and GlobalMood datasets.
    • Achieved superior accuracy and Macro-F1 scores, demonstrating effectiveness in music emotion recognition.
    • Ablation studies confirmed the significant contributions of the global Transformer encoder, harmonic-aware attention, and CHA learning objective.
    • The framework demonstrated a favorable balance between performance and computational complexity.

    Conclusions:

    • The novel deep learning framework offers a robust and scalable solution for multimodal music emotion recognition.
    • The study advances hierarchical modeling and harmonic-aware representation learning in affective computing.
    • The proposed methods effectively capture complex musical structures and emotional nuances from multimodal data.