Seen and Heard

April 1, 2008
Read my lips: Edgar Bergen and Charlie McCarthy

Read my lips: Edgar Bergen and Charlie McCarthy
Bettman / CORBIS

The prevailing wisdom among brain scientists has been that each of the five senses—sight, hearing, smell, touch, and taste—is governed by its own corresponding region of the brain," says Jennifer Groh, associate professor of psychology and neuroscience. "The view has been that each of these areas processes the information separately and sends that information to the cortex, which puts it all together at the end."

But a recent study conducted by Groh and a team from her multisensory research lab offers surprising new insights into how the brain processes a multitude of stimuli from the outside world. By studying monkeys, the researchers found that information from the eyes and ears is actually processed together before the combined signals make it to the cortex.

Groh is especially interested in a tiny round structure in the brain known as the inferior colliculus. This structure, less than a half-inch in diameter, is located in the most primitive area of the brain. It is one of several early stops in the brain for signals leaving the ear, headed for the cortex.

"In our experiments, we found that this structure, which had been assumed to mainly process auditory information, actually responds to visual information as well," Groh says. "In fact, about 64 percent of the neurons in the inferior colliculus can carry visual as well as auditory signals. This means that visual and auditory information gets combined quite early, and before the 'thinking part' of the brain can make sense of it."

That is why ventriloquism seems to work, she says. The association between the voice and the moving mouth of the dummy is made before the viewer consciously thinks about it. The same process may also explain why the words being spoken by a talking head on television appear to be coming out of the mouth, even though the television speakers are located to the side of the set.

"The eyes see the lips moving, and the ears hear the sound, and the brain immediately jumps to the conclusion about the origin of the voice," she says.

The results of the experiments were published online in the Proceedings of the National Academy of Sciences. Groh and her team are now conducting experiments to determine whether one of the senses influences how the other is perceived.