In the first experiment the two experimental conditions involved audio-visual speech using the original video (AV) and a modified version of the video which only conveys temporal information (AV*). The results of the AV conditions were first published by Ross et al. (2007). Noise levels were varied and recognition performance was recorded. In the natural AV condition maximal improvement was obtained at noise levels with a signal to noise ratio of -12dB. These are some examples of stimuli without noise, and with noise at -12dB:
Examples of congruent AV stimuli:
no noise:
beg,
count,
cup
-12dB noise:
boot,
dish
Examples of congruent AV* stimuli (modified video):
no noise:
barge,
book,
nine
-12dB noise:
cheap,
cute
The main contribution of this paper is a theoretical model that describes the improvements observed resulting from showing the video information for both these conditions. The model is based on the idea that subjects integrate information in a statistical optimal fashion. To confirm the model we used it to make predictions on the answers subjects would give if the audio and video stimuli did not match. The experiments confirmed the model's prediction. Here are some examples of these incongruent stimuli:
Examples of incongruent AV stimuli:
no noise:
V: care, A: tear
-12dB noise:
V: bay, A: gay
V: care, A: tear
V: fame, A: flame
Original video shown in congruent-AV condition is the same as in Ross et al. 2007, Cerebral Cortex 17, pp. 1147-1153. Video shown in congruent-AV* condition was generated using softwared provided Tue Lehn-Schioler (see his Ph.D. Thesis 2005). Incongruent AV condition was generated from the original using conventional video editing tools.
Lucas C. Parra, March 4, 2009