Regular articleNeural correlates of incongruous visual information: An event-related fMRI study
Introduction
Incongruity is the quality generated by the violation of norms and expectations built up over a lifetime. Multiple behavioral studies have shown a memory advantage for incongruous versus commonplace material (e.g., Riefer and LaMay, 1998). This result is commonly known as the “bizarreness effect.” To provide a better understanding of the neural correlates of this effect, we conducted a functional magnetic resonance imaging (fMRI) experiment directly contrasting responses evoked by incongruous versus ordinary visual stimuli. A corresponding memory advantage was demonstrated for incongruous over ordinary stimuli in a behavioral recognition experiment.
Previous studies have established that semantically incongruous sentences (e.g., “The soldier licked the kittens”) are better recalled than ordinary sentences (e.g., “The man read a book”) Cornoldi et al 1988, Hirshman et al 1989, McDaniel and Einstein 1986, Nicolas and Marchal 1996, Riefer and LaMay 1998, Worthen and Marshall 1996. Incongruity in these studies typically is induced by violating semantic consistency, for example, by attributing human actions to animals and artifacts or vice versa. Subjects typically are asked to read incongruous and ordinary sentences with attention to semantic content. A memory advantage on subsequent testing for incongruous material has so far been reliably demonstrated mostly with recall (as opposed to recognition) tasks and with mixed (including incongruous and ordinary items) rather than pure lists.
Several mutually nonexclusive interpretations of the bizarreness effect are currently under debate (see, e.g., Worthen et al., 2000). One possibility is that incongruous stimuli elicit extra (more elaborated) processing because they are more difficult to make sense of in the context of expected properties and semantic norms (the elaboration hypothesis) Merry 1980, Wollen and Cox 1981. According to this view, the extra elaboration given to incongruous items strengthens their memory trace much as deep, semantically elaborated tasks encourage memorization in standard memory paradigms (e.g., Craik and Lockhart 1972, Craik and Tulving 1975. Related to this is the finding that the memory benefits of incongruity are due in part to distinctiveness in the encoding context Hunt and Elliot 1980, McDaniel and Einstein 1986. Another possibility is that incongruous stimuli elicit a surprise response due to the violation of expectations, which enhances contextual cues available for later recall (the surprise hypothesis; Hirshman et al., 1989).
The physiological basis of the bizarreness effect is incompletely understood. Relevant data derive from studies of electrophysiological responses to incongruous stimuli. The N400 potential is a central-parietal scalp negativity that peaks 400 ms after the presentation of (1) semantically incongruous words in the context of sentences (see Kutas and Hillyard, 1983), or (2) semantically unrelated words or pictures (see Holcomb, 1988, and McPherson and Holcomb, 1999). The N400 phenomenon is thought to be a correlate of the semantic conflict arising when perceived information is different from information expected based on prior context. Three recent fMRI studies Kuperberg et al 2000, Newman et al 2001, Ni et al 2001 have compared responses to semantic (e.g., “The man sailed the hotel to China”) versus syntactic (e.g., “The woman read the with letter attention”) anomalies. All three studies report functional anatomic differences in responses to semantic versus syntactic anomalies, but, in aggregate, do not provide a consistent picture of the specifically activated regions.
The physiological correlates of the surprise response proposed by Hirshman et al. (1989) have not been defined. The “surprise” may originate in an orienting response, which is usually evoked by unexpected stimuli and corresponds to a complex of processes including arousal, capture of attention, and memory formation (Sokolov, 1963). The electrophysiological phenomenon most associated with orienting is the P3 potential, i.e., scalp positivity peaking about 300 ms after infrequent (oddball) stimuli are presented Johnson 1986, Johnson 1993, Matt et al 1992. Two P3 subtypes may be distinguished: Whereas the P3b is strongly modulated by stimulus task relevance and does not habituate, the P3a occurs in response to unexpected or context-violating events, exhibits a somewhat earlier latency and more anterior scalp distribution, and habituates rapidly Knight and Nakada 1998, Knight and Scabini 1998. Several fMRI correlates of the P3a have recently been reported suggesting that the hippocampus as well as the temporal, prefrontal, and parietal cortices contribute to the generation of this potential Kirino et al 2000, Knight 1996, McCarthy et al 1997, Strange et al 2000.
In a series of two studies, we explored the neural and behavioral correlates of incongruous visual material to test whether such stimuli elicit extra elaboration and/or a surprise response. Incongruous pictures incorporating semantic violations concerning artifacts and living things were generated by combining incongruous parts (e.g., head of cat on body of lobster, kangaroo with the head replaced by the hose/nozzle assembly of a gasoline pump). One condition was designed to examine the response to incongruous materials under conditions usually used in previous behavioral memory experiments. In this condition (frequent incongruous, or Freq-Inc condition), incongruous and ordinary pictures were presented with equal frequency. Such a condition was expected to allow examination of additional encoding activity (if any) related to incongruity. Another condition (infrequent incongruous, or Inf-Inc condition) was designed to examine the surprise response (if any) associated with the perception of incongruous pictures. We tried to maximize the possibility of a surprise response by (1) presenting only few incongruous pictures among several ordinary pictures, and (2) always running this condition first so that these incongruous pictures were the first seen in the experiment. A small number of oddball stimuli (color inverted but otherwise ordinary pictures) were also included as a control for frequency effects.
Although firm predictions are not possible, the elaboration hypothesis tentatively predicts greater activity for incongruous pictures than for ordinary pictures in areas underlying the processing of these stimuli. This extra processing should occur independently of the frequency of the incongruous pictures. Accordingly, data supporting this hypothesis should show greater activity for incongruous pictures across our conditions in visual extrastriate cortex and prefrontal cortex, reflecting visual and semantic encoding (see Buckner et al 1999, Fletcher et al 1998, Tulving et al 1994. The surprise hypothesis predicts greater activity for incongruous pictures than for ordinary pictures in areas underlying emotional responses such as the thalamus, the amygdala, and/or the frontal operculum (see Strange et al., 2000). This emotional response should be enhanced (or only appear) when incongruous stimuli are novel and infrequent (Inf-Inc condition).
In addition to the fMRI study, we conducted a separate experiment to examine the effect of stimulus incongruity on recognition memory. The study phase of this behavioral experiment was closely matched to the Freq-Inc condition of the fMRI experiment. A novel sequential recognition design was implemented that provided sensitive measures of recognition accuracy as well as a measure of recognition confidence. We hypothesized that incongruous stimuli would be recognized more accurately and more quickly than ordinary stimuli.
Section snippets
Participants
Twenty-four participants (13 females, mean age = 21.8 years) were recruited from the Washington University community. All participants had normal or corrected-to-normal vision, were native English speakers, showed a strong right-handed preference as measured by the Edinburg Handedness Inventory (Raczkowski et al., 1974), and reported no history of significant neurological problems. Participants were paid and provided informed consent in accordance with guidelines set by the Washington
Experiment 2: recognition memory for incongruous pictures
Experiment 2 was designed to assess recognition memory for the stimuli used in Experiment 1. To exclude frequency effects it was necessary to study a group of participants separate from those in the fMRI experiment. In accordance with standard practice, both the study and test phases of Experiment 2 included equal numbers of incongruous and ordinary pictures (as in the fMRI Freq-Inc condition). A recognition memory design with a novel response method was used.
General discussion
Neural and behavioral correlates of visual incongruity were explored to better understand how such information is processed and why it is remembered better than ordinary information. First, an extensive network of regions was differentially activated by incongruous pictures both during frequent and infrequent presentation contexts. Regions modulated by incongruous pictures spanned the ventral and dorsal visual pathways bilaterally, the frontal cortex along the inferior frontal gyrus, parietal
Acknowledgements
Support was provided by grants from the James S. McDonnell Foundation. We thank Laura E. Williams for her help during the scanning sessions.
References (72)
- et al.
Dissociating working memory from task difficulty in human prefrontal cortex
Neuropsychologia
(1997) - et al.
Levels of processinga framework for memory research
J. Verb. Learn. Behav.
(1972) - et al.
Functional MRI studies of spatial and nonspatial working memory
Cogn. Brain Res.
(1998) - et al.
The novelty P3an event-related brain potential (ERP) sign of the brain’s evaluation of novelty
Neurosci. Behav. Rev.
(2001) - et al.
An investigation of paradoxical memory effects
J. Memory Lang.
(1989) Automatic and attentional processingan event-related brain potential analysis of semantic priming
Brain Lang.
(1988)- et al.
Recognizing words, pictures, and conceptsa comparison of lexical, object, and reality decisions
J. Verb. Learn. Verb. Behav.
(1984) - et al.
Characterizing the hemodynamic responseeffects of presentation rate, sampling procedure, and the possibility of ordering brain activity based on relative timing
Neuroimage
(2000) - et al.
Event-related brain potentials during initial encoding and recognition memory of congruous and incongruous words
J. Memory Lang.
(1986) - et al.
Anatomic localization and quantitative analysis of gradient refocused echo-planar fMRI susceptibility artifacts
Neuroimage
(1997)
Reliability and validity of some handedness questionnaire items
Neurospsychologia
Event-related potentials and the recollection of associative information
Cogn. Brain Res.
Brain mechanisms for detecting perceptual, semantic and emotional deviance
Neuroimage
Event-related potentials during discourse-level semantic integration of complex pictures
Cogn. Brain Res.
Bizarreness versus integration of mental images as determinants of learning
Cogn. Psychol.
Processing strategies for time-course data sets in functional MRI of the human brain
Magn. Reson. Med.
Making memoriesbrain activity that predicts how well visual experience will be remembered
Science
Frontal cortex contributes to human memory formation
Nat. Neurosci.
Randomized event-related experimental designs allow for extremely rapid presentation rates using functional MRI
Neuroreport
Psyscopean interactive graphic system for designing and controlling experiments in the psychology laboratory using Macintosh computers
Behav. Res. Methods Instr. Computers
The influence of the nature of material and of mental operations on the occurrence of the bizarreness effect
Q. J. Exp. Psychol.
Transient and sustained activity in a distributed neural system for human working memory
Nature
Depth of processing and the retention of words in episodic memory
J. Exp. Psychol. Gen.
Bizarreness effects in verbal tasks and subject-performed tasks
Eur. J. Cogn. Psych.
P300 and recall in an incidental memory paradigm
Psychophysiology
The neural consequences of conflict between intention and the senses
Brain
The functional roles of prefrontal cortex in episodic memory. I. Encoding
Brain
The search for “common sense:”an electrophysiological study of the comprehension of words and pictures in reading
J. Cogn. Neurosci.
Perceptual differentiation and category effects in normal object recognition. A PET study
Brain
Searching for a baselinefunctional imaging and the resting human brain
Nat. Rev. Neurosci.
The role of nonsemantic information in memoryorthographic distinctiveness effects upon retention
J. Exp. Psychol. Gen.
A triarchic model of P300 amplitude
Psychophysiology
On the neural generators of the P300 component of the event-related potential
Psychophysiology
Neural sources involved in auditory target detection and novelty processingan event-related fMRI study
Psychophysiology
Prefrontal activation evoked by infrequent target and novel stimuli in a visual target detection taskan event-related functional magnetic resonance imaging study
J. Neurosci.
Cited by (84)
Effect of the congruity of emotional contexts at encoding on source memory: Evidence from ERPs
2022, International Journal of PsychophysiologyMemory influences visual cognition across multiple functional states of interactive cortical dynamics
2019, Psychology of Learning and Motivation - Advances in Research and TheoryCitation Excerpt :Key additional evidence comes from “pseudo-object” effects, in which pictures of real objects are compared to pictures of pseudo-objects, such as objects with novel features, novel spatial configurations, and real objects combined into a novel figure, and people perform a variety of tasks, such as object or reality decisions, category or memory decisions, or passively view the pictures (e.g., Kroll & Potter, 1984). In ERP and fMRI studies, the N3 complex is more negative and the PHT networks are more active for pseudo-objects than real objects (Barbarotto, Laiacona, Macchi, & Capitani, 2002; Daffner, Mesulam, Scinto, Calvo, et al., 2000; Ganis et al., 2007; Gruber & Muller, 2005, 2006; Holcomb & McPherson, 1994; McPherson & Holcomb, 1999; Michelon, Snyder, Buckner, McAvoy, & Zacks, 2003; Schendan & Ganis, 2015; Schendan et al., 1998). Crucially, this is the same pattern as for impoverishment effects, that is, comparisons between more than less impoverished real objects.
Auditory conflict and congruence in frontotemporal dementia
2017, NeuropsychologiaCitation Excerpt :These neuroanatomical substrates are in line with our experimental hypotheses and with previous neuroanatomical work in auditory and other modalities. Processing of both semantic and emotional auditory congruence had substrates in anterior temporal and insula cortices that are likely to constitute ‘hubs’ for processing signal patterns and salient deviations based on prior expectations or stored templates (Christensen et al., 2011; Clark et al., 2015b; Gauvin et al., 2016; Groussard et al., 2010; Merkel et al., 2015; Michelon et al., 2003; Nazimek et al., 2013; Remy et al., 2014; Watanabe et al., 2014). These regions are engaged during matching of incoming signals against previously learned semantic and affective schemas (Groussard et al., 2010; Zahn et al., 2009).