when we hear with our eyes

Visual and auditory information is essential for understanding speech. When we talk to someone, we not only hear what they say, but we also watch how they say it by moving their lips.

The ability of human speech is based on the integration of visual and auditory information, which is demonstrated by the fact that we can experience a curious illusory phenomenon: the McGurk effect.

It can be said that this particular phenomenon occurs when we hear with our eyes, causing what we hear to change according to what we see. Let’s see what this interesting visual-auditory effect is all about.

What is the McGurk effect?

We tend to believe that our senses work independently: when we hear, we only hear; and when we see, we only see. Based on this belief, it would be reasonable to think that a visual stimulus is not capable of distorting the way we perceive sound. However, the reality is that it can, since our perceptual experiences are the product of a complex process of mixing information, the same mixing that gives rise to a particular phenomenon: the McGurk effect.

Surely you have had a conversation more than once in an extremely noisy environment. Maybe it was in a nightclub, on the terrace of a bar on a busy street, or in a high school classroom. When there is a loud background noise, it is difficult for us to understand what the person in front of us is saying, and to understand something we use the old instinctive trick of watching his mouth as he speaks.

In these cases, visual and auditory information is not analyzed separately, but rather is combined. The human brain has a region called the superior temporal sulcus, which is specialized in the combination of the two types of information, in the examples that we have given it would be responsible for combining the phonemes that our interlocutor speaks with the movement of his lips.

Thanks to this ability to combine multimodal information, the superior temporal sulcus is the neurological stage where the illusory McGurk effect occurs, which would be nothing more than the result of an error in the decoding of the message when two different sensory modalities interact, causing what we see does not coincide with what we hear.

If we do a quick search on YouTube, we can find more than one video where this phenomenon is exposed in a practical way. This link takes us to a good example of this phenomenon:

In this precise case, the person in the video says / ba / all the time, however, depending on how they move their lips, you can hear either / ba / or / pa /.

This effect can also occur with other combinations of syllables. For example, this can be achieved with the combination / ka / (visual) and / pa / (auditory), resulting in the perception of / ta /. Another example would be seeing someone making lip movements that match the syllable / ga / but while the syllable / ba / is being spoken, it will be perceived as / da /.

The way we hear the same sound varies greatly depending on whether or not we watch the way the person speaking to us moves their lips. This not only affects the perception of simple sets of sounds such as syllables, but it has also been proven to work with full sentences, although you have surely witnessed this yourself in some of the situations we have mentioned earlier. .

One of the first discoveries related to the McGurk effect and the interplay between sensory modalities is that having the possibility of seeing the lips of our interlocutor move considerably improves the volume of what we hear.

We have seen that it gives us the sensation of hearing phonemes up to 15 decibels louder when we have the transmitter in our visual field. This happens even when the acoustic conditions are not unfavorable, such as being in a room without sound or in a quiet place.

History of its discovery

This phenomenon was first described in 1976 in an article by British cognitive psychologist Harry McGurk with his colleague John MacDonald titled “Hearing Lips and Seeing Voices”. Their original study involved studying the imitation patterns of a group of children who were developing the ability to speak, and the experiment involved showing them several videos of people saying different syllables.

However, a read error has occurred. The technician in charge of editing the video made a mistake and caused the picture and sound to be out of sync, causing the recording of a person to appear saying something that did not match the sound heard .

As the video was playing, McGurk and MacDonald heard a third phoneme instead of the one spoken with the lips and the one that was uttered. It was a happy coincidence that these two researchers discovered this particular auditory illusion.

Its importance in the study of human speech

The discovery of this effect is considered proof that the visual and auditory systems have evolved together to allow, among other things, better speech processing. Our visual system helps us discriminate between sounds that are difficult to tell apart, a benefit deaf people realize when they lip read.

Being able to see how our interlocutor moves their lips increases confidence in the message perceived through the auditory system. That is, if two independent systems point to the same solution, in this case the same message, we trust that message more than if we only receive it through one channel.

It is to highlight that the McGurk effect does not happen automatically. For this to happen, we need to be attentive to our interlocutor, and when distracting stimuli are incorporated, both visual and auditory, this illusion is lessened.

In fact, this proves that the effect is not due to poor reception of visual or auditory information, but to an error in the integration of these two sensory modalities.

Another fact that gives strength to the idea that the visual system supports the auditory is that when we see a person speaking to us but cannot hear what they are saying to us at all, not only our visual cortex is activated, but also the auditory one, even if we don’t listen to anything.

The McGurk effect and brain dysfunction

We saw that brain damage and reading disorders, in addition to manifesting mental disorders, it influences the likelihood of the McGurk effect occurring.

People who have had a callosotomy show the McGurk effect more slowly. It appears that children with specific language impairments have the McGurk effect weaker compared to children who do not have language acquisition disorders or reading and writing difficulties.

Laterality also influences, as right-handed people are more likely to experience this effect.

The McGurk effect in different languages

Regardless of the language spoken, its speakers depend to some extent on visual information when perceiving speech. However, the intensity of the McGurk effect varies from one language to anotherWe see that in languages ​​such as Spanish, Turkish, Italian, Dutch, English and German, their speakers feel this effect more strongly than Chinese and Japanese speakers.

The fact that speakers of Asian languages ​​show the McGurk effect less frequently may be due to the cultural practice of avoiding eye contact.. Moreover, Chinese and Japanese in particular are two languages ​​with very syllabic linguistic structures, usually of consonant + vowel and consonant + vowel + consonant type, which makes them particularly adept at detecting syllables no matter which way you move. the words. interlocutor.