Related Experiment Video
Updated: Jun 29, 2026

Development of an Audio-based Virtual Gaming Environment to Assist with Navigation Skills in the Blind
Published on: March 27, 2013
Enhancing Video Experiences for DHH Individuals Through Sound-Inspired Motion Caption-Based Spatiotemporal Tacton
None:
When deaf and hard of hearing (DHH) individuals watch videos, captions are essential for them to understand the linguistic content. Current captions, however, are not suitable for conveying non-verbal sound information, such as background music, sound effects, or speech nuances. In this paper, we designed a multimodal system, Motion Caption Haptic System (MCHS), that enables DHH individuals to encounter sounds in videos through animated caption and spatiotemporal vibration patterns, supporting a more vivid and immersive experience. We elaborately designed motion captions and spatiotemporal haptic patterns for representative sound effects and spoken emotions to work well together through surveys from 27 DHH and 64 hearing participants. An evaluation with 19 DHH individuals demonstrated the capabilities and potential of the MCHS to improve their video viewing experience, along with a discussion of important issues that need to be addressed when designing multimodal captioning systems for the DHH viewers.
More Related Videos
05:43Applying Incongruent Visual-Tactile Stimuli during Object Transfer with Vibro-Tactile Feedback
Published on: May 23, 2019
11:39Assessment of Audio-Tactile Sensory Substitution Training in Participants with Profound Deafness Using the Event-Related Potential Technique
Published on: September 7, 2022
Related Concept Videos
Perception of Sound Waves
The pitch of a sound depends on the frequency and the pressure amplitude of the source. Two sounds of the same frequency...
Sound as Pressure Waves
The pressure fluctuation depends on the difference in displacements between the successive points in the...
Relative Motion Analysis using Rotating Axes
However, to express the relative position of point B relative to point A, an additional frame of reference, denoted as x'y', is necessary. This additional frame not only translates but also rotates relative to the fixed frame, making it instrumental in...
Relative Motion Analysis - Acceleration
Relative Motion Analysis using Rotating Axes - Acceleration
Time differentiation is...
Perceiving Loudness, Pitch, and Location
Place theory, or place coding, suggests that different pitches are heard because various sound waves activate specific locations along the cochlea's basilar membrane. The brain determines the pitch of a sound by identifying...