Search research articles

ABOUT JoVE

Overview Leadership Blog JoVE Help Center

AUTHORS

Publishing Process Editorial Board Scope & Policies Peer Review FAQ Submit

LIBRARIANS

Testimonials Subscriptions Access Resources Library Advisory Board FAQ

RESEARCH

JoVE Journal Methods Collections JoVE Encyclopedia of Experiments Archive

EDUCATION

JoVE Core JoVE Business JoVE Science Education JoVE Lab Manual Faculty Resource Center Faculty Site

Terms & Conditions of Use

Related Concept Videos

Sensory Modalities

Sensory Modalities

Sensation typically is the process by which the sensory receptors and sense organs detect stimuli from the internal and external environment and transmit this information to the central nervous system for processing.
General senses refer to the broad category of sensory information detected by receptors in the body and can be further grouped into somatic and visceral senses. Somatic sensations include touch, pressure, temperature, and pain and are essential for navigating our environment and...

You might also read

Related Articles

Articles linked to this work by shared authors, journal, and citation graph.

Sort by

Same author

Genetic architecture of the limbic white matter microstructure in aging and Alzheimer's Disease.

Alzheimer's & dementia : the journal of the Alzheimer's Association·2026

Same author

Is correction for gradient nonlinearity necessary in a brain diffusion tensor MRI clinical study?

PloS one·2026

Same author

Unified bilayer membranes with mechanically reinforced interface for stage-adaptive bone regeneration.

Nature communications·2026

Same author

Charting Cervical Spinal Cord Morphometry Across the Lifespan.

bioRxiv : the preprint server for biology·2026

Same author

Analytic Bounds on GAMLSS Model Variability of Normative White Matter Brain Charts.

Proceedings of SPIE--the International Society for Optical Engineering·2026

Same author

Lifespan Trajectories of Asymmetry in White Matter Tracts.

Human brain mapping·2026

Search research articles

Related Experiment Video

Updated: Oct 11, 2025

A Dual Task Procedure Combined with Rapid Serial Visual Presentation to Test Attentional Blink for Nontargets

A Dual Task Procedure Combined with Rapid Serial Visual Presentation to Test Attentional Blink for Nontargets

Published on: December 5, 2014

Structured Multimodal Attentions for TextVQA.

Chenyu Gao, Qi Zhu, Peng Wang

IEEE Transactions on Pattern Analysis and Machine Intelligence

|December 2, 2021

Summary

This summary is machine-generated.

This study introduces a Structured Multimodal Attention (SMA) neural network for Text-based Visual Question Answering (TextVQA). SMA significantly improves performance by enhancing text reading and multimodal reasoning, outperforming state-of-the-art models.

More Related Videos

Methods to Test Visual Attention Online

Methods to Test Visual Attention Online

Published on: February 19, 2015

Eye Tracking During Visually Situated Language Comprehension: Flexibility and Limitations in Uncovering Visual Context Effects

Eye Tracking During Visually Situated Language Comprehension: Flexibility and Limitations in Uncovering Visual Context Effects

Published on: November 30, 2018

Related Experiment Videos

Last Updated: Oct 11, 2025

A Dual Task Procedure Combined with Rapid Serial Visual Presentation to Test Attentional Blink for Nontargets

A Dual Task Procedure Combined with Rapid Serial Visual Presentation to Test Attentional Blink for Nontargets

Published on: December 5, 2014

Methods to Test Visual Attention Online

Methods to Test Visual Attention Online

Published on: February 19, 2015

Eye Tracking During Visually Situated Language Comprehension: Flexibility and Limitations in Uncovering Visual Context Effects

Eye Tracking During Visually Situated Language Comprehension: Flexibility and Limitations in Uncovering Visual Context Effects

Published on: November 30, 2018

Area of Science:

Computer Science
Artificial Intelligence
Machine Learning

Background:

Text-based Visual Question Answering (TextVQA) presents a challenge requiring models to interpret image text and answer questions.
Existing state-of-the-art (SoTA) Visual Question Answering (VQA) models struggle with TextVQA due to limitations in text recognition and multimodal reasoning.

Purpose of the Study:

To propose an end-to-end Structured Multimodal Attention (SMA) neural network to address text reading and textual-visual reasoning deficiencies in TextVQA.
To enhance the performance of models on TextVQA benchmarks by improving their ability to process and reason over visual text.

Main Methods:

Developed an SMA neural network utilizing a structural graph representation to encode image relationships (object-object, object-text, text-text).
Employed a multimodal graph attention network for reasoning over the encoded relationships.
Integrated a global-local attentional answering module for iterative answer generation by combining Optical Character Recognition (OCR) tokens and general vocabulary, following the M4C approach.

Main Results:

The proposed SMA model demonstrated superior performance on the TextVQA dataset and two tasks of the ST-VQA dataset, surpassing SoTA models (excluding pre-trained TAP).
The model achieved first place in the TextVQA Challenge 2020, highlighting its strong reasoning capabilities.
Investigated the impact of OCR performance on VQA accuracy, showing that while all models benefit, SMA shows the most significant improvement due to its robust textual-visual reasoning.

Conclusions:

The SMA model effectively addresses key challenges in TextVQA, particularly in text reading and multimodal reasoning.
The study provides valuable insights into the relationship between OCR accuracy and TextVQA performance.
Released human-annotated ground-truth OCR annotations for TextVQA to facilitate future research and establish a fair testing baseline.