Jove
Visualize
Contact Us
JoVE
x logofacebook logolinkedin logoyoutube logo
ABOUT JoVE
OverviewLeadershipBlogJoVE Help Center
AUTHORS
Publishing ProcessEditorial BoardScope & PoliciesPeer ReviewFAQSubmit
LIBRARIANS
TestimonialsSubscriptionsAccessResourcesLibrary Advisory BoardFAQ
RESEARCH
JoVE JournalMethods CollectionsJoVE Encyclopedia of ExperimentsArchive
EDUCATION
JoVE CoreJoVE BusinessJoVE Science EducationJoVE Lab ManualFaculty Resource CenterFaculty Site
Terms & Conditions of Use
Privacy Policy
Policies

Related Concept Videos

Labeling Emotion01:20

Labeling Emotion

397
Emotional labeling is a cognitive process that involves identifying and naming one's emotions, such as anger, fear, happiness, or sadness. It allows individuals to recognize and express their internal emotional states, a critical aspect of emotional regulation and communication. Labeling emotions requires more than mere recognition; it also involves drawing upon memory and contextual cues to understand the current situation and apply a corresponding emotional label. For instance, feeling...
397
Emotional Expression01:26

Emotional Expression

575
Emotional expression encompasses how individuals convey their emotions through verbal communication and non-verbal cues. These non-verbal actions include facial expressions, body language, and physical gestures, such as frowning or smiling. Among these, facial expressions play a crucial role in emotional expression and are understood universally, indicating a biological basis for how humans communicate emotions.
Universal Facial Expressions
Psychologist Paul Ekman identified seven basic...
575
Physiology of Emotion01:20

Physiology of Emotion

2.0K
The physiology of emotions is a multifaceted process involving the autonomic nervous system, brain structures, hormones, and neurotransmitters. This intricate interplay dictates how emotions manifest in the body and influence behavior.
Autonomic Nervous System
The autonomic nervous system (ANS) plays a critical role in emotional responses by regulating involuntary physiological functions. It consists of two main components: the sympathetic and parasympathetic systems. The sympathetic system...
2.0K
Facial Feedback Hypothesis01:24

Facial Feedback Hypothesis

336
Charles Darwin proposed that facial expressions are an evolutionary adaptation for communication. He argued that these expressions are not influenced by culture but are universal across species. For example, a snarling expression with exposed teeth signals a threat in many animals, including humans. Darwin also suggested that displaying an emotion can intensify the feeling. Smiling, for example, could enhance one's sense of happiness. This idea laid the foundation for understanding the role...
336
Non-Verbal Cues01:29

Non-Verbal Cues

76
Non-verbal communication extends beyond gestures and facial expressions to include vocal elements known as paralanguage. Paralanguage consists of non-verbal vocal cues such as pitch, loudness, speech rate, pauses, and non-verbal vocalizations like laughter, sighs, and moans. These elements not only accompany speech but also provide critical emotional and contextual information.The Role of Paralanguage in CommunicationParalanguage adds depth to spoken language by conveying emotions and...
76
Force Classification01:22

Force Classification

1.9K
Forces play a crucial role in the study of physics and engineering. They are essential in describing the motion, behavior, and equilibrium of objects in the physical world. Forces can be classified based on their origin, type, and direction of action.
Contact and non-contact forces are two of the most widely used categories of forces. As the name suggests, contact forces require physical contact between two objects to act upon each other. Examples of contact forces include frictional,...
1.9K

You might also read

Related Articles

Articles linked to this work by shared authors, journal, and citation graph.

Sort by
Same author

[Identification of Placenta hominis and its adulterants using COI barcode].

Zhongguo Zhong yao za zhi = Zhongguo zhongyao zazhi = China journal of Chinese materia medica·2014
Same author

Two new species of Austrophthiracarus (Acari: Oribatida: phthiracaridae) from New Zealand.

Zootaxa·2014
Same author

The genus Notophthiracarus of New Zealand (Acari: Oribatida: Phthiracaridae): three new species and a key to 24 described species.

Zootaxa·2014
Same author

MHC class II restricted innate-like double negative T cells contribute to optimal primary and secondary immunity to Leishmania major.

PLoS pathogens·2014
Same author

Hepatic perfusion parameters of contrast-enhanced ultrasonography correlate with the severity of chronic liver disease.

Ultrasound in medicine & biology·2014
Same author

Dietary accumulation of tetrabromobisphenol A and its effects on the scallop Chlamys farreri.

Comparative biochemistry and physiology. Toxicology & pharmacology : CBP·2014

Related Experiment Video

Updated: Oct 26, 2025

Author Spotlight: Addressing Technical and Subjective Challenges in Measuring Classroom Attention
06:37

Author Spotlight: Addressing Technical and Subjective Challenges in Measuring Classroom Attention

Published on: December 15, 2023

4.6K

Multi-Modal Fusion Emotion Recognition Method of Speech Expression Based on Deep Learning.

Dong Liu1, Zhiyong Wang1, Lifeng Wang1

  • 1School of Information Engineering, Shandong Youth University of Political Science, Jinan, China.

Frontiers in Neurorobotics
|July 26, 2021
PubMed
Summary

This study introduces a deep learning method for multimodal emotion recognition, combining voice and facial expressions. The approach achieves high accuracy, outperforming other methods on benchmark datasets.

Keywords:
LibSVM classifierdeep learningemotion recognitionexpressionlong short-term memorymultimodal fusionvoice

More Related Videos

Author Spotlight: Investigating the Impact of Emotional Prosodies on Voice Recognition and Perception
05:48

Author Spotlight: Investigating the Impact of Emotional Prosodies on Voice Recognition and Perception

Published on: August 9, 2024

1.7K

Related Experiment Videos

Last Updated: Oct 26, 2025

Author Spotlight: Addressing Technical and Subjective Challenges in Measuring Classroom Attention
06:37

Author Spotlight: Addressing Technical and Subjective Challenges in Measuring Classroom Attention

Published on: December 15, 2023

4.6K
Author Spotlight: Investigating the Impact of Emotional Prosodies on Voice Recognition and Perception
05:48

Author Spotlight: Investigating the Impact of Emotional Prosodies on Voice Recognition and Perception

Published on: August 9, 2024

1.7K

Area of Science:

  • Artificial Intelligence
  • Computer Vision
  • Speech Processing

Background:

  • Traditional emotion recognition methods struggle with redundant information and noise from single-modal feature extraction.
  • Deep learning offers potential for improved performance in complex tasks like emotion recognition.

Purpose of the Study:

  • To propose a novel deep learning-based multimodal fusion method for accurate emotion recognition from speech and facial expressions.
  • To address limitations of single-modal approaches by integrating diverse data streams.

Main Methods:

  • Utilized Convolutional Neural Network-Long Short-Term Memory (CNN-LSTM) for voice feature extraction.
  • Employed Inception-ResNet-v2 for facial expression feature extraction from video.
  • Applied Long Short-Term Memory (LSTM) for inter- and intra-modal correlation analysis.
  • Integrated feature selection via chi-square test and classifier LIBSVM for final emotion recognition.

Main Results:

  • Achieved 87.56% accuracy on the MOSI dataset.
  • Achieved 90.06% accuracy on the MELD dataset.
  • Demonstrated superior performance compared to existing methods.

Conclusions:

  • The proposed multimodal fusion method significantly enhances emotion recognition accuracy.
  • This research provides a theoretical foundation for applying multimodal fusion in emotion recognition systems.