Jove
Visualize
Contact Us
JoVE
x logofacebook logolinkedin logoyoutube logo
ABOUT JoVE
OverviewLeadershipBlogJoVE Help Center
AUTHORS
Publishing ProcessEditorial BoardScope & PoliciesPeer ReviewFAQSubmit
LIBRARIANS
TestimonialsSubscriptionsAccessResourcesLibrary Advisory BoardFAQ
RESEARCH
JoVE JournalMethods CollectionsJoVE Encyclopedia of ExperimentsArchive
EDUCATION
JoVE CoreJoVE BusinessJoVE Science EducationJoVE Lab ManualFaculty Resource CenterFaculty Site
Terms & Conditions of Use
Privacy Policy
Policies

Related Concept Videos

Vision01:24

Vision

56.0K
Vision is the result of light being detected and transduced into neural signals by the retina of the eye. This information is then further analyzed and interpreted by the brain. First, light enters the front of the eye and is focused by the cornea and lens onto the retina—a thin sheet of neural tissue lining the back of the eye. Because of refraction through the convex lens of the eye, images are projected onto the retina upside-down and reversed.
56.0K
Neural Circuits01:25

Neural Circuits

1.8K
Neural circuits and neuronal pools are two of the main structures found in the nervous system. Neural circuits are networks of neurons that work together to carry out a specific task or process. They consist of interconnected neurons and glial cells, which provide structural and metabolic support.
Neuronal pools are collections of nerve cells with similar functions and interact through chemical and electrical signals. These pools include both interneurons (the central neural circuit nodes that...
1.8K
Visual System01:26

Visual System

826
Light enters the eye through the cornea, a transparent, dome-shaped surface covering the surface of the eyeball that helps to direct and focus incoming light. This light is then channeled toward the pupil, an adjustable opening whose size is controlled by the iris. The iris, a pigmented muscle, regulates the amount of light entering the eye by contracting or dilating the pupil, thereby ensuring optimal light levels for clear vision.
Once through the pupil, the light passes through the lens, a...
826

You might also read

Related Articles

Articles linked to this work by shared authors, journal, and citation graph.

Sort by
Same author

Expression of tissue factor pathway inhibitor-2 in gastric stromal tumor and its clinical significance.

Experimental and therapeutic medicine·2014
Same author

Facile access to cytocompatible multicompartment micelles with adjustable Janus-cores from A-block-B-graft-C terpolymers prepared by combination of ROP and ATRP.

Colloids and surfaces. B, Biointerfaces·2014
Same author

Functional layers for Zn(II) ion detection: from molecular design to optical fiber sensors.

The journal of physical chemistry. B·2013
Same author

Expression of the 78 kD glucose-regulated protein is induced by endoplasmic reticulum stress in the development of hepatopulmonary syndrome.

Gene·2013
Same author

Multi-nuclear silver(I) and copper(I) complexes: a novel bonding mode for bispyridylpyrrolides.

Dalton transactions (Cambridge, England : 2003)·2013
Same author

Transcriptome profilings of female Schistosoma japonicum reveal significant differential expression of genes after pairing.

Parasitology research·2013

Related Experiment Video

Updated: Oct 8, 2025

A Swin Transformer-Based Model for Thyroid Nodule Detection in Ultrasound Images
04:23

A Swin Transformer-Based Model for Thyroid Nodule Detection in Ultrasound Images

Published on: April 21, 2023

2.0K

A CNN-transformer hybrid approach for decoding visual neural activity into text.

Jiang Zhang1, Chen Li1, Ganwanming Liu1

  • 1College of Electrical Engineering, Sichuan University, Chengdu 610065, China.

Computer Methods and Programs in Biomedicine
|December 28, 2021
PubMed
Summary

This study introduces a novel CNN-Transformer model to translate visual brain activity from natural images into descriptive text, advancing our understanding of visual processing.

Keywords:
Brain decodingCNNDeep learningTransformerfMRI

More Related Videos

Swin-PSAxialNet: An Efficient Multi-Organ Segmentation Technique
04:48

Swin-PSAxialNet: An Efficient Multi-Organ Segmentation Technique

Published on: July 5, 2024

548
Author Spotlight: Enhancement of Salient Object Detection for Smart Grid Applications
03:31

Author Spotlight: Enhancement of Salient Object Detection for Smart Grid Applications

Published on: December 15, 2023

676

Related Experiment Videos

Last Updated: Oct 8, 2025

A Swin Transformer-Based Model for Thyroid Nodule Detection in Ultrasound Images
04:23

A Swin Transformer-Based Model for Thyroid Nodule Detection in Ultrasound Images

Published on: April 21, 2023

2.0K
Swin-PSAxialNet: An Efficient Multi-Organ Segmentation Technique
04:48

Swin-PSAxialNet: An Efficient Multi-Organ Segmentation Technique

Published on: July 5, 2024

548
Author Spotlight: Enhancement of Salient Object Detection for Smart Grid Applications
03:31

Author Spotlight: Enhancement of Salient Object Detection for Smart Grid Applications

Published on: December 15, 2023

676

Area of Science:

  • Neuroscience
  • Computer Vision
  • Natural Language Processing

Background:

  • Current research primarily decodes language structure from linguistic stimuli.
  • The human brain frequently processes non-linguistic stimuli like natural images.
  • A model is needed to map visual neural activity from images to content.

Purpose of the Study:

  • To develop an end-to-end model for decoding visual neural activity evoked by natural images.
  • To create descriptive text from functional magnetic resonance imaging (fMRI) signals.

Main Methods:

  • A hybrid Convolutional Neural Network (CNN)-Transformer model was designed.
  • The model encodes semantic sequences from fMRI signals using a 1D CNN.
  • It then decodes the representation into English sentences.

Main Results:

  • Decoded texts showed semantic consistency with ground truth annotations.
  • Specific Transformer architectures were found to be optimal for this task.
  • The model successfully translated visual neural activity into descriptive text.

Conclusions:

  • The proposed model can decode visual neural activity from natural images into descriptive sentences.
  • This approach offers a potential tool for neuroscientists studying visual information processing.
  • It aids in understanding the neural mechanisms of visual perception.