Search research articles

ABOUT JoVE

Overview Leadership Blog JoVE Help Center

AUTHORS

Publishing Process Editorial Board Scope & Policies Peer Review FAQ Submit

LIBRARIANS

Testimonials Subscriptions Access Resources Library Advisory Board FAQ

RESEARCH

JoVE Journal Methods Collections JoVE Encyclopedia of Experiments Archive

EDUCATION

JoVE Core JoVE Business JoVE Science Education JoVE Lab Manual Faculty Resource Center Faculty Site

Terms & Conditions of Use

Related Concept Videos

You might also read

Related Articles

Articles linked to this work by shared authors, journal, and citation graph.

Sort by

Same author

Evaluation of Parkinson's Disease Motor Symptoms via Wearable Inertial Measurements Units and Surface Electromyography Sensors.

Bioengineering (Basel, Switzerland)·2025

Same author

Automated UPDRS Gait Scoring Using Wearable Sensor Fusion and Deep Learning.

Bioengineering (Basel, Switzerland)·2025

Same author

Sliding Mode Repetitive Control Based on the Unknown Dynamics Estimator of a Two-Stage Supply Pressure Hydraulic Hexapod Robot.

Biomimetics (Basel, Switzerland)·2025

Same author

A phase division-based multi-segment foot model for estimating dynamic foot arch stiffness during walking.

PloS one·2025

Same author

Influence of Sampling Rate on Wearable IMU Orientation Estimation Accuracy for Human Movement Analysis.

Sensors (Basel, Switzerland)·2025

Same author

Endpoint Distribution Modeling-Based Capture Algorithm for Interfering Multi-Target.

Sensors (Basel, Switzerland)·2025

Same journal

RETRACTED: Zhang et al. A Novel Framework for Reconstruction and Imaging of Target Scattering Centers via Wide-Angle Incidence in Radar Networks. <i>Sensors</i> 2025, <i>25</i>, 6802.

Sensors (Basel, Switzerland)·2026

Same journal

Enhancing Unsupervised Multi-Source Domain Adaptation for Person Re-Identification via Mixture of Experts and Graph-Based Relation.

Sensors (Basel, Switzerland)·2026

Same journal

Development of an Instrumented Glove for Palmar Pressure Assessment in Kayakers.

Sensors (Basel, Switzerland)·2026

Same journal

Development and Experimental Validation of an Autonomous IoT-Based Monitoring System for Real-Time Water Quality Assessment in the Amazon River.

Sensors (Basel, Switzerland)·2026

Same journal

Semi-Supervised Adversarial Learning Framework for Controller Area Network Bus Intrusion Detection.

Sensors (Basel, Switzerland)·2026

Same journal

Smart Optimization Method for Safety Signs in Innovative Manufacturing Environments Integrating Industrial Field IoT Sensors and Knowledge Graphs.

Sensors (Basel, Switzerland)·2026

See all related articles

Search research articles

Related Experiment Video

Updated: May 10, 2025

A Swin Transformer-Based Model for Thyroid Nodule Detection in Ultrasound Images

A Swin Transformer-Based Model for Thyroid Nodule Detection in Ultrasound Images

Published on: April 21, 2023

A Novel Approach for Visual Speech Recognition Using the Partition-Time Masking and Swin Transformer 3D Convolutional

Xiangliang Zhang¹, Yu Hu², Xiangzhi Liu¹

¹The State Key Laboratory of Fluid Power and Mechatronic Systems, School of Mechanical Engineering, Zhejiang University, Hangzhou 310027, China.

Sensors (Basel, Switzerland)

|April 26, 2025

Summary

This summary is machine-generated.

This study introduces Partition-Time Masking (PTM), a novel data augmentation technique to enhance visual speech recognition. PTM improves lip reading model performance and generalization, especially in challenging conditions.

Keywords:

data augmentation deep learning human–computer interaction lip reading recognition algorithms

More Related Videos

Swin-PSAxialNet: An Efficient Multi-Organ Segmentation Technique

Swin-PSAxialNet: An Efficient Multi-Organ Segmentation Technique

Published on: July 5, 2024

Author Spotlight: Investigating the Impact of Emotional Prosodies on Voice Recognition and Perception

Author Spotlight: Investigating the Impact of Emotional Prosodies on Voice Recognition and Perception

Published on: August 9, 2024

Related Experiment Videos

Last Updated: May 10, 2025

A Swin Transformer-Based Model for Thyroid Nodule Detection in Ultrasound Images

A Swin Transformer-Based Model for Thyroid Nodule Detection in Ultrasound Images

Published on: April 21, 2023

Swin-PSAxialNet: An Efficient Multi-Organ Segmentation Technique

Swin-PSAxialNet: An Efficient Multi-Organ Segmentation Technique

Published on: July 5, 2024

Author Spotlight: Investigating the Impact of Emotional Prosodies on Voice Recognition and Perception

Author Spotlight: Investigating the Impact of Emotional Prosodies on Voice Recognition and Perception

Published on: August 9, 2024

Area of Science:

Computer Science
Artificial Intelligence
Machine Learning

Background:

Visual speech recognition (lip reading) offers advantages in noisy environments and for individuals with speech impairments.
Current lip reading models struggle with generalization, confusable phonemes, lighting variations, and facial occlusions.

Purpose of the Study:

To introduce Partition-Time Masking (PTM), a data augmentation method for improving lip reading models.
To propose a novel lip reading recognition architecture, Swin Transformer and 3D Convolution (ST3D).

Main Methods:

Implemented Partition-Time Masking (PTM) for data augmentation using nonlinear transformations.
Developed the Swin Transformer and 3D Convolution (ST3D) model architecture, combining Swin Transformer with 3D convolution.
Validated PTM with the DC-TCN model on the LRW dataset and ST3D on LRW and LRW1000 datasets.

Main Results:

The PTM method achieved a peak accuracy of 92.15% on the LRW dataset using the DC-TCN model.
The ST3D model reached 91.8% accuracy on the LRW dataset and 56.1% on the challenging LRW1000 dataset.
The proposed methods demonstrated superior performance compared to current mainstream lip reading models, particularly on easily confused samples.

Conclusions:

Partition-Time Masking (PTM) effectively enhances the generalization ability and performance of lip reading models.
The ST3D architecture shows significant improvements in visual speech recognition, outperforming existing models.
The research addresses key challenges in lip reading, paving the way for more robust and accurate systems.