Related Experiment Video
Updated: Aug 27, 2025
07:52
Investigating Motor Skill Learning Processes with a Robotic Manipulandum
Published on: February 12, 2017
8.8K
Robot gaining robust pouring skills through fusing vision and audio
Zhongli Wang1, Guohui Tian1, Hao Pan1
1School of Control Science and Engineering, Shandong University, 17923 Jingshi Road, Jinan, China.
ISA Transactions
|September 29, 2022
Summary
This study introduces a novel visual-audio fusion network for service robots to accurately estimate liquid height during pouring tasks. This multimodal approach enhances robot pouring skills by combining visual and auditory data for precise control.
Area of Science:
- Robotics
- Artificial Intelligence
- Sensor Fusion
Background:
- Accurate liquid height estimation is critical for service robot pouring tasks.
- Individual vision or audio sensors provide insufficient accuracy for robust liquid estimation.
- Existing methods lack the precision required for complex pouring operations.
Purpose of the Study:
- To develop a multimodal visual-audio information fusion network for enhanced robot pouring.
- To improve the accuracy and robustness of liquid height and cup state estimation.
- To enable service robots with superior pouring capabilities.
Main Methods:
- Visual features extracted using a residual network with an attention model.
- Audio features extracted via Fast Fourier Transform (FFT) and Long Short-Term Memory (LSTM).
- Fusion of visual and audio features using a fully connected network for outputting liquid height and cup state.
- A sinusoidal and transient fusion control method for gripper angle output.
Main Results:
- The proposed visual-audio fusion network significantly improves liquid height estimation accuracy.
- The system effectively estimates the state of the cup during the pouring process.
- Experimental validation confirms the algorithm's effectiveness in service robot pouring tasks.
Conclusions:
- Multimodal sensor fusion is superior to single-modality sensing for robot pouring tasks.
- The developed network provides a robust solution for accurate liquid estimation and control.
- This approach enables service robots to perform pouring tasks with enhanced precision and skill.
Related Concept Videos
01:24Vision
55.1K
Vision is the result of light being detected and transduced into neural signals by the retina of the eye. This information is then further analyzed and interpreted by the brain. First, light enters the front of the eye and is focused by the cornea and lens onto the retina—a thin sheet of neural tissue lining the back of the eye. Because of refraction through the convex lens of the eye, images are projected onto the retina upside-down and reversed.
55.1K
01:19Vibrating Concrete
168
Mechanical vibrators are instrumental in compacting newly poured concrete within formwork and around reinforcements. This process is essential to eliminate trapped air pockets and establish a dense concrete mass. One widely used method is vibrating by internal vibrators, often referred to as a poker vibrator or immersion vibrator. It is rapidly inserted through the full depth of the freshly laid concrete and slightly extends into the layer below it (which remains in a plastic state). Consistent...
168
01:08Brick Cutting Techniques
223
Brick-cutting techniques involve various tools and methods to shape bricks for construction. A mason's hammer with a chisel-pointed end is used for basic shaping through sharp, precise strikes. For more complex shapes requiring higher precision, a power saw with a water-cooled diamond blade is used.
Cut bricks are categorized by size. Bricks cut to half their original length are called half-bats, while those cut to three-fourths their length are known as three-fourth bats.
Special types of...
Cut bricks are categorized by size. Bricks cut to half their original length are called half-bats, while those cut to three-fourths their length are known as three-fourth bats.
Special types of...
223
01:22Mechanical Systems
265
Mechanical systems are analogous to to electrical networks where springs and masses play similar roles to inductors and capacitors, respectively. A viscous damper in mechanical systems functions similarly to a resistor in electrical networks, dissipating energy. The forces acting on a mass in such systems include an applied force in the direction of motion, counteracted by forces from the spring, a viscous damper, and the mass's acceleration. This interplay of forces is mathematically...
265
01:21Perceiving Loudness, Pitch, and Location
405
The human brain perceives pitch through two primary mechanisms reflected in place theory and frequency theory. Each mechanism describes how sound waves are interpreted as specific pitches by the brain, offering insights into the intricate processes of auditory perception.
Place theory, or place coding, suggests that different pitches are heard because various sound waves activate specific locations along the cochlea's basilar membrane. The brain determines the pitch of a sound by...
Place theory, or place coding, suggests that different pitches are heard because various sound waves activate specific locations along the cochlea's basilar membrane. The brain determines the pitch of a sound by...
405
01:24Muscle Coordination and Action
1.9K
Muscle coordination is a complex and finely tuned process essential for smooth and purposeful movements like flexion, extension, adduction, abduction, and rotation. The human body orchestrates the actions of various muscles working in concert, each with a specific role. Four functional types describe how muscles work together: agonist, antagonist, synergist, and fixator.
Agonists
Agonist muscles, often called prime movers, are the primary muscles responsible for producing a specific movement....
Agonists
Agonist muscles, often called prime movers, are the primary muscles responsible for producing a specific movement....
1.9K