Automatic lipreading

We have implemented an automatic visual lipreading system intended to supplement a standard automatic speech recognizer. The system extracts a small set of parameters from a video image sequence. These include geometrical and local brightness information about the images. Standard ASR techniques are applied to the data. Performance is tested on a small task of discriminating 22 consonants, with and without added white noise. The visual system alone achieves approximately 47% accuracy. More importantly, however, when it is used in conjunction with an audio-based speech recognizer, the performance of the combined system exceeds that of either system used alone. This effect is more pronounced when white noise is added to the audio signal, and can result in as much as a 61% error reduction depending on the noise level.