Related Experiment Videos
Aroma: Mixed-Initiative AI Assistance for Non-Visual Cooking by Grounding Multimodal Information Between Reality and
Zheng Ning1, Leyang Li1, Daniel Killough2
1Department of Computer Science and Engineering, University of Notre Dame, Notre Dame, Indiana, USA.
Proceedings of the ACM Symposium on User Interface Software and Technology. ACM Symposium on User Interface Software and Technology
|July 23, 2026
Summary
A new AI system, Aroma, helps blind or low-vision (BLV) individuals cook by integrating non-visual cues with video recipes. This system provides real-time, context-aware assistance for daily living activities.
Area of Science:
- Human-Computer Interaction
- Assistive Technology
- Artificial Intelligence
Background:
- Audiovisual video recipes are inaccessible to blind or low-vision (BLV) individuals.
- BLV individuals often rely on non-visual sensory cues for daily living activities like cooking.
- Existing video recipes do not cater to the unique needs of BLV users.
Purpose of the Study:
- To introduce Aroma, an AI system designed to assist BLV individuals in following video recipes.
- To explore a mixed-initiative approach for real-time, context-aware assistance.
- To enhance the accessibility of daily living activities for BLV individuals.
Main Methods:
- Developed Aroma, an AI system integrating user-perceived non-visual cues, a wearable camera feed, and video recipe content.
- Implemented a mixed-initiative approach for proactive and responsive user assistance.
- Conducted a study with eight BLV participants to evaluate Aroma's effectiveness.
Main Results:
- Aroma successfully provided timely, context-aware assistance to BLV users.
- The system helped align the user's physical environment with video recipe instructions.
- Participants could better interpret their current state and follow recipe steps with Aroma's support.
Conclusions:
- Aroma demonstrates the potential of AI systems to support BLV individuals in performing activities of daily living.
- Integrating non-visual cues and AI offers a promising direction for accessible technology design.
- Further research into interactive AI systems for BLV users is warranted.
Related Concept Videos
01:15Sensory Modalities
Sensation typically is the process by which the sensory receptors and sense organs detect stimuli from the internal and external environment and transmit this information to the central nervous system for processing.
General senses refer to the broad category of sensory information detected by receptors in the body and can be further grouped into somatic and visceral senses. Somatic sensations include touch, pressure, temperature, and pain and are essential for navigating our environment and...
General senses refer to the broad category of sensory information detected by receptors in the body and can be further grouped into somatic and visceral senses. Somatic sensations include touch, pressure, temperature, and pain and are essential for navigating our environment and...
01:27Tactile and Chemical Senses
Tactile senses encompass touch, temperature, and pain, each mediated by specific receptors. Touch receptors detect mechanical energy or pressure against the skin. Sensory fibers from these receptors enter the spinal cord and relay information to the brain stem. Here, most fibers cross over to the opposite side of the brain. The touch information then moves to the thalamus, which projects a map of the body's surface onto the somatosensory areas of the parietal lobes in the cerebral cortex. This...
01:28Automatic Processing and Automatic Social Behavior
Automatic processing refers to the cognitive operations that occur without conscious intent or awareness, playing a fundamental role in shaping social cognition and behavior. These processes enable individuals to navigate complex social environments efficiently by relying on mental shortcuts and pre-existing knowledge structures known as schemas. One of the most influential mechanisms underlying automatic processing is priming, which subtly activates mental representations through exposure to...
01:25Olfaction
The sense of smell is achieved through the activities of the olfactory system. It starts when an airborne odorant enters the nasal cavity and reaches olfactory epithelium (OE). The OE is protected by a thin layer of mucus, which also serves the purpose of dissolving more complex compounds into simpler chemical odorants. The size of the OE and the density of sensory neurons varies among species; in humans, the OE is only about 9-10 cm2.
The olfactory receptors are embedded in the cilia of the...
The olfactory receptors are embedded in the cilia of the...
01:28Perception
Perception is a fundamental psychological process that enables individuals to organize, interpret, and consciously experience sensory information. This process is crucial for understanding and interacting with the world around us. It includes both bottom-up and top-down processing, each playing a distinct role in how we perceive our environment.
Bottom-up processing begins at the sensory level, where receptors detect external environmental stimuli. These could include the tactile sensation of...
Bottom-up processing begins at the sensory level, where receptors detect external environmental stimuli. These could include the tactile sensation of...
01:12Visual Agnosia
Visual agnosia is a condition characterized by the inability to recognize visually presented objects despite having normal vision. For instance, a person with visual agnosia can describe the shape and color of an object but cannot identify or name it. This impairment does not affect their visual field, acuity, color vision, brightness discrimination, language, or memory. An example of this condition in a social setting is someone at a dinner party asking for "that silver thing with a round end"...