Stop Talking, Start Influencing

2.

Images + Speech

We hear and apprehend only what we already half know.

- Henry David Thoreau

ABBA.

Dancing Queen.

I take little pride in admitting it, but for a short period during my formative teenage years, this was easily my favourite song. In fact, I'd say I listened to it upwards of 300 times throughout the mid-90s - and during all that time I was absolutely certain that the opening lyrics were: 'You can dance, you can die, having the time of your life'. To be honest, this did always strike me as slightly macabre, but come on . it's ABBA . what else could I expect?

Jump to earlier this year. I'm at work surfing YouTube when I randomly stumble upon the original Dancing Queen music video. Despite my long history with this song, I had only ever heard ABBA sing it - now, I would actually be able to watch ABBA sing it.

Fully expecting to be transported to my glorious youth, I hit play . that's when I saw it. During the opening verse, both singers clearly rounded their pursed lips into an unmistakable 'J' form. For the first time in my life, I heard the correct lyrics: 'You can dance, you can jive'. The song was never about someone passing away; it's always been about someone getting down (which, in hindsight, makes a whole lot more sense).

So what happened? How is it possible that watching someone sing a song could overwrite twenty years of memory forged by listening to that song?

Hear with your eyes, see with your ears

My Dancing Queen debacle was a real-world example of the McGurk Effect, a psychological phenomenon that illustrates how what we see can drive what we hear.

If you were to take part in a typical McGurk experiment, it would go something like this. You're sitting in front of a computer screen, watching a man exaggeratedly mouth the word 'baba' while over a set of speakers you hear him saying that same word. Every second or so you hear it: 'baba . baba . baba'.

Suddenly, while the voice continues unchanged over the speakers, the man on the screen starts mouthing a different word. Rather than pressing his lips together to form 'ba', he deliberately puts his front teeth against his lower lip and exaggeratedly makes a 'fa' form.

That's when it happens.

Rather than the clear 'b' syllable you've been hearing all along, you start to hear a much softer 'f'. Even though you know the audio hasn't changed in any way, you start to hear a totally different word: 'fafa . fafa . fafa'.

ILLUSTRATION 10. THE MCGURK EFFECT

Convinced the researchers must be playing some sort of trick on you, you close your eyes. With the face gone, the audio clearly reverts to what it's been all along: 'baba'. But as soon as you open your eyes and focus back on the man's face, the voice shifts again to 'fafa'.

Many people seem willing to accept that vision can drive hearing (perhaps this has to do with the tangible nature of visual objects as opposed to the largely 'invisible' nature of sound waves). However, it might come as a bit of a shock to learn that this relationship isn't a one-way street. For as many instances as we can find of vision driving hearing, there are an equal number of hearing driving vision.

Perhaps the best-known example is something known as the Shams Illusion. If you were to take part in a typical Shams experiment, it would start similarly to the McGurk experiment above. You're sitting in front of a blank computer screen. Suddenly, at random intervals, a loud beep plays over the pair of speakers while, simultaneously, a small circle quickly appears and disappears on the screen.

Occasionally, two loud beeps play in rapid succession over the speakers while two circles quickly appear and disappear on the screen. It's all fairly simple - when you hear one beep, you see one circle; when you hear two beeps, you see two circles. Nothing to get too excited about.

Of course, it's all an illusion. Regardless of how many beeps are played over the speakers, only one circle is ever flashed on the computer screen. At no time do two circles ever appear on the screen - yet you'd swear you see two circles every time you hear two beeps. This is essentially the flipside of the McGurk Effect. Whereas before what you saw changed what you heard, in this illusion what you hear changes what you see.

ILLUSTRATION 11. YOUR VISUAL BRAIN

ILLUSTRATION 12. SIGHT AND SOUND MIX FREELY - NO BOTTLENECK!

The ecological shift

There's clearly a strong and integrated relationship between what we see and what we hear. But how does all of this work?

In the last chapter we learnt that when we hear something, this information is first processed along the sides of the brain within the auditory cortex.

Conversely, when we see something, this information is first processed along the back of the brain within the visual cortex. This very large neural area is divided into several distinct regions, each of which serves to process a different aspect of whatever it is we are looking at. For instance, when watching a bird fly, different regions of the visual cortex process the edges of the bird, the colour of the bird, the motion of the bird, etc.

Earlier, we saw that trying to funnel two different streams of information down a single processing channel causes a bottleneck which leads to a loss of information. Luckily, hearing and vision utilize different processing channels. This not only eliminates any bottleneck, but also allows us to combine hearing and vision into a single, consolidated signal. This process is what is meant by sensory integration.

Importantly, sensory integration is not an additive process (A + B = A and B) - it is an ecological process (A + B = C). As an example, imagine if you dropped a dozen non-native beetles into an otherwise balanced garden. The results wouldn't simply be the same garden plus beetles. Rather, this one addition would change everything: the food chain, the nutrients in the soil, the survival conditions. The same is true for hearing and vision. In a very real sense, when what we hear joins with what we see, a completely new entity emerges. Perhaps no process has better epitomized the phrase 'The whole is greater than the sum of its parts'.

Let's see what this all means in the real world.

Every Saturday night, three good friends get together. When Jerry and Casey arrive at Karen's house, Karen is sitting in her room writing some notes. She quickly gathers the cards and stands up to greet her friends at the door. They follow her into the living room but, as usual, they can't agree on exactly what to play. Casey eventually takes a stand and they begin. Early in the evening, Casey notices Karen's hand and the many diamonds. As the night progresses, the tempo of play increases. Finally, Karen says 'Let's hear the score.' They listen carefully and comment on their performance. When all is done, Karen's friends go home, exhausted but happy.

ILLUSTRATION 13. INTERPRETATION - PART I
(INSPIRED BY ANDERSON ET AL. 1977)

Interpretation

Let me quickly acknowledge the elephant in the room: over the next couple of sections I am going to attempt to demonstrate how hearing and vision combine to generate meaning. To that end, I'm not 100 per cent sure a book is the best medium with which to accomplish this task. However, as we learnt in the previous chapter, reading written text is similar to listening to oral speech. As such, although the examples we'll use aren't ideal, they should be effective enough to get the basic points across.

To begin, I'd like you to take a look at the picture opposite, then read the passage following it.

Again, allowing for the fact that reading is standing in for listening, I imagine you found this passage relatively simple and straightforward. It's an innocuous tale of a group of friends gathering for a weekend card game. Fair enough.

Now I'd like you to take a look at the image on the next page and read the passage following it.

Same story, same words, same sounds entering your ears - yet the accompanying image changes how you interpret this auditory information. When the image changes, words like notes, score and performance suddenly take on a totally different meaning. Similarly, phrases like 'She quickly gathers the cards and stands up' and 'the tempo of play increases' resonate in completely novel ways. Importantly, this interaction works both ways: the sounds you are hearing (or, in this case, reading) impact the way you interpret each visual image. In the first sequence, you likely attached the name Casey to one of the male characters and took special interest in Karen's flush draw. Conversely, in the second sequence, you likely attached the same name to one of the female characters and took special interest in the rings adorning Karen's bow hand.

This is what I meant earlier by an ecological process. As information enters your eyes, it changes the way you process and interpret information entering your ears. Similarly, as information enters your ears, it changes the way you process and interpret information entering your eyes. The whole is greater than the sum of its parts.

Every Saturday night, three good friends get together. When Jerry and Casey arrive at Karen's house, Karen is sitting in her room writing some notes. She quickly...