End-to-end audio-visual learning for cochlear implant sound coding simulations in noisy environments.

Area of Science:

• Biomedical Engineering
• Artificial Intelligence
• Auditory Neuroscience

Background:

• Cochlear implants (CI) restore hearing for severe-to-profound hearing loss via electrical stimulation.
• Listening in noisy environments remains a significant challenge for CI users.
• Deep learning offers new approaches for CI sound coding, particularly by integrating visual information.

Purpose of the Study:

• To develop and evaluate an end-to-end CI system integrating an audio-visual speech enhancement (AVSE) module with the ElectrodeNet-CS (ECS) model.
• To assess the system's performance in improving speech intelligibility and signal quality in simulated noisy conditions.

Main Methods:

• An AVSE module was integrated with the ECS model to create the AVSE-ECS system.
• The AVSE-ECS system underwent joint training to optimize its performance.
• Objective speech intelligibility metrics and signal-to-error ratio (SER) were used for evaluation.

Main Results:

• The AVSE-ECS system demonstrated high objective speech intelligibility.
• A significant improvement in signal-to-error ratio (SER) of 7.4666 dB was achieved compared to advanced combination encoder strategies.
• Joint training of the AVSE-ECS system yielded superior results.

Conclusions:

• The integration of AVSE modules shows significant promise for advancing CI sound coding strategies.
• This audio-visual approach offers a potential solution to improve speech perception in noise for CI users.
• The AVSE-ECS system represents a step forward in developing more effective cochlear implant technologies.