Neural correlates of incongruous visual information

Abstract

Incongruous information is better remembered than ordinary information. This result has been attributed both to semantic incongruity and surprise. To determine the contribution of each factor, we performed a functional magnetic resonance imaging study in which participants viewed pictures depicting ordinary and incongruous objects (e.g., head of a wrench fused onto a sheep body). To maximize surprise we administered novel incongruent pictures infrequently in an initial scan. (This scan also included infrequent color-inverted pictures as a control for frequency.) To obtain a pure measure of the effect of incongruity we conducted a second scan in which participants viewed equal numbers of ordinary and incongruous pictures. Signal increases were greater for incongruous versus ordinary and oddball stimuli throughout the ventral and dorsal visual pathways, and in prefrontal cortex bilaterally. Signal decreases were larger for incongruous than for ordinary stimuli bilaterally in lateral parietal regions. A subset of regions near the right frontal operculum and extending laterally responded only to, or more strongly to, infrequent incongruous pictures. A second, purely behavioral, experiment involving a separate group of participants demonstrated that incongruous pictures were better recognized than ordinary pictures. We interpret our results as suggesting that, although correlates of a surprise response can be observed, better memory for incongruous visual information is attributable mainly to more processing and, consequently, better encoding.

Introduction

Incongruity is the quality generated by the violation of norms and expectations built up over a lifetime. Multiple behavioral studies have shown a memory advantage for incongruous versus commonplace material (e.g., Riefer and LaMay, 1998). This result is commonly known as the “bizarreness effect.” To provide a better understanding of the neural correlates of this effect, we conducted a functional magnetic resonance imaging (fMRI) experiment directly contrasting responses evoked by incongruous versus ordinary visual stimuli. A corresponding memory advantage was demonstrated for incongruous over ordinary stimuli in a behavioral recognition experiment.

Previous studies have established that semantically incongruous sentences (e.g., “The soldier licked the kittens”) are better recalled than ordinary sentences (e.g., “The man read a book”) Cornoldi et al 1988, Hirshman et al 1989, McDaniel and Einstein 1986, Nicolas and Marchal 1996, Riefer and LaMay 1998, Worthen and Marshall 1996. Incongruity in these studies typically is induced by violating semantic consistency, for example, by attributing human actions to animals and artifacts or vice versa. Subjects typically are asked to read incongruous and ordinary sentences with attention to semantic content. A memory advantage on subsequent testing for incongruous material has so far been reliably demonstrated mostly with recall (as opposed to recognition) tasks and with mixed (including incongruous and ordinary items) rather than pure lists.

Several mutually nonexclusive interpretations of the bizarreness effect are currently under debate (see, e.g., Worthen et al., 2000). One possibility is that incongruous stimuli elicit extra (more elaborated) processing because they are more difficult to make sense of in the context of expected properties and semantic norms (the elaboration hypothesis) Merry 1980, Wollen and Cox 1981. According to this view, the extra elaboration given to incongruous items strengthens their memory trace much as deep, semantically elaborated tasks encourage memorization in standard memory paradigms (e.g., Craik and Lockhart 1972, Craik and Tulving 1975. Related to this is the finding that the memory benefits of incongruity are due in part to distinctiveness in the encoding context Hunt and Elliot 1980, McDaniel and Einstein 1986. Another possibility is that incongruous stimuli elicit a surprise response due to the violation of expectations, which enhances contextual cues available for later recall (the surprise hypothesis; Hirshman et al., 1989).

The physiological basis of the bizarreness effect is incompletely understood. Relevant data derive from studies of electrophysiological responses to incongruous stimuli. The N400 potential is a central-parietal scalp negativity that peaks 400 ms after the presentation of (1) semantically incongruous words in the context of sentences (see Kutas and Hillyard, 1983), or (2) semantically unrelated words or pictures (see Holcomb, 1988, and McPherson and Holcomb, 1999). The N400 phenomenon is thought to be a correlate of the semantic conflict arising when perceived information is different from information expected based on prior context. Three recent fMRI studies Kuperberg et al 2000, Newman et al 2001, Ni et al 2001 have compared responses to semantic (e.g., “The man sailed the hotel to China”) versus syntactic (e.g., “The woman read the with letter attention”) anomalies. All three studies report functional anatomic differences in responses to semantic versus syntactic anomalies, but, in aggregate, do not provide a consistent picture of the specifically activated regions.

The physiological correlates of the surprise response proposed by Hirshman et al. (1989) have not been defined. The “surprise” may originate in an orienting response, which is usually evoked by unexpected stimuli and corresponds to a complex of processes including arousal, capture of attention, and memory formation (Sokolov, 1963). The electrophysiological phenomenon most associated with orienting is the P3 potential, i.e., scalp positivity peaking about 300 ms after infrequent (oddball) stimuli are presented Johnson 1986, Johnson 1993, Matt et al 1992. Two P3 subtypes may be distinguished: Whereas the P3b is strongly modulated by stimulus task relevance and does not habituate, the P3a occurs in response to unexpected or context-violating events, exhibits a somewhat earlier latency and more anterior scalp distribution, and habituates rapidly Knight and Nakada 1998, Knight and Scabini 1998. Several fMRI correlates of the P3a have recently been reported suggesting that the hippocampus as well as the temporal, prefrontal, and parietal cortices contribute to the generation of this potential Kirino et al 2000, Knight 1996, McCarthy et al 1997, Strange et al 2000.

In a series of two studies, we explored the neural and behavioral correlates of incongruous visual material to test whether such stimuli elicit extra elaboration and/or a surprise response. Incongruous pictures incorporating semantic violations concerning artifacts and living things were generated by combining incongruous parts (e.g., head of cat on body of lobster, kangaroo with the head replaced by the hose/nozzle assembly of a gasoline pump). One condition was designed to examine the response to incongruous materials under conditions usually used in previous behavioral memory experiments. In this condition (frequent incongruous, or Freq-Inc condition), incongruous and ordinary pictures were presented with equal frequency. Such a condition was expected to allow examination of additional encoding activity (if any) related to incongruity. Another condition (infrequent incongruous, or Inf-Inc condition) was designed to examine the surprise response (if any) associated with the perception of incongruous pictures. We tried to maximize the possibility of a surprise response by (1) presenting only few incongruous pictures among several ordinary pictures, and (2) always running this condition first so that these incongruous pictures were the first seen in the experiment. A small number of oddball stimuli (color inverted but otherwise ordinary pictures) were also included as a control for frequency effects.

Although firm predictions are not possible, the elaboration hypothesis tentatively predicts greater activity for incongruous pictures than for ordinary pictures in areas underlying the processing of these stimuli. This extra processing should occur independently of the frequency of the incongruous pictures. Accordingly, data supporting this hypothesis should show greater activity for incongruous pictures across our conditions in visual extrastriate cortex and prefrontal cortex, reflecting visual and semantic encoding (see Buckner et al 1999, Fletcher et al 1998, Tulving et al 1994. The surprise hypothesis predicts greater activity for incongruous pictures than for ordinary pictures in areas underlying emotional responses such as the thalamus, the amygdala, and/or the frontal operculum (see Strange et al., 2000). This emotional response should be enhanced (or only appear) when incongruous stimuli are novel and infrequent (Inf-Inc condition).

In addition to the fMRI study, we conducted a separate experiment to examine the effect of stimulus incongruity on recognition memory. The study phase of this behavioral experiment was closely matched to the Freq-Inc condition of the fMRI experiment. A novel sequential recognition design was implemented that provided sensitive measures of recognition accuracy as well as a measure of recognition confidence. We hypothesized that incongruous stimuli would be recognized more accurately and more quickly than ordinary stimuli.