Study shows how brain interprets surfaces

The stucco-like surface on the right looks darker and glossier than the one on the left even though the mean luminance of both images is equal. The luminance histogram of the image on the right is positively skewed and the one on the left is negatively skewed.
MIT researchers and colleagues investigating how the brain interprets the appearance of surfaces think they have an answer. They have found that the perception of reflectance and gloss are correlated with certain statistical properties of the image. These properties could be coded by neurons that respond differentially to light and dark spots.


The research team, a collaboration between MIT and the NTT Communication Science Labs in Japan, reported its findings in the April 18 online issue of Nature.

Studying how the brain analyzes surface appearance is not only important to understanding the workings of the human brain, but could also help scientists develop better visual systems for robots.

"We know a lot about the perception of objects, but much less about the perception of the materials that the objects are made of," said Edward Adelson, an author of the paper and professor of visual science in the Department of Brain and Cognitive Sciences at MIT. "Our studies show that statistical skewness, which quantifies an asymmetry between light and dark patterns, has a strong influence on the way a material is perceived."

Analyzing visual attributes such as color, texture, and gloss, is critical in everyday tasks such as deciding whether a patch of pavement is icy, whether a pancake is cooked, or whether skin is healthy, according to the researchers.

In their experiments, the research team asked subjects to rate the lightness and glossiness of natural materials such as stucco or fabric. The surfaces have a mix of dark shadows and bright highlights, giving rise to distinctive visual patterns.

"Natural surfaces are complicated," said Adelson. "They have bumps and dips, and the light reflects in complex ways, producing characteristic statistical patterns." These patterns serve as signatures both for the shape and the material composing the surface.

The researchers quantified the images in terms of "luminance histograms," which plot the distribution of pixel values. They also estimated the histograms of responses of neurons that respond to light and dark spots in an image. In both cases, they found that the "skewness" of the histogram, which measures its asymmetry, was correlated with the subject's perceptions of surface qualities. Positive skewness led to darker and glossier surface appearance.
The researchers also found that they could manipulate subjects' perceptions of glossiness by digitally manipulating the skewness of the images, said study author Lavanya Sharan, an MIT graduate student at the Computer Science and Artificial Intelligence Lab.

Technology based on this research could also be useful for autonomous vehicles, said Sharan.

"You want to know what kind of surface you're on--is the road dry or wet or icy? Are you on a dirt road? A machine vision system needs to make these judgments based on the surface appearance," she said.

The exact neural mechanism for detecting image skewness is unknown, but the researchers have offered a tentative model.

In the retina and brain, there are cells that are preferentially sensitive to either bright patterns or dark ones. After pooling responses within each cell population, the brain could compare the balance between very bright and very dark patterns. This balance determines the skewness.

"We don't know what the underlying physiology is, but what we do know is that the computation of skewness is something that could be done very easily with the hardware that does exist in the early parts of the brain's visual system," said Adelson.

The researcher's skewness model is supported by experiments with visual aftereffects, which showed that the human visual system can adapt to skewness. Similar to what happens when you stare at an image of a certain color and then see an after-image, staring at patterns with a high degree of positive skewness will cause the next pattern you look at to appear negatively skewed, which causes the material to appear lighter and less glossy.


The lead author on the Nature paper is Isamu Motoyoshi of the Human and Information Science Lab at NTT Communication Science Labs in Japan. Shin'ya Nishida of NTT is also an author on the paper.