Ch t 10Ch t 10Chapter 10.Chapter 10.
C iti N i Th Bi l f th Mi d 3C iti N i Th Bi l f th Mi d 3rdrd EdEdCognitive Neuroscience: The Biology of the Mind, 3Cognitive Neuroscience: The Biology of the Mind, 3rdrd Ed.,Ed.,
M. S. Gazzaniga, R. B. Ivry, and G. R. Mangun, Norton, 2008.M. S. Gazzaniga, R. B. Ivry, and G. R. Mangun, Norton, 2008.
H. Y. Kim, E. S. Lee, and B.-T. Zhang
Biointelligence Laboratory, Seoul National University
I t d tiI t d tiIntroductionIntroduction
“Hi, Jack!” vs. “Hijack!”
Human language’s complexityHuman language s complexity
Introduction of a cognitive neuroscience
i h h lperspective on the human language system
(C) 2009, SNU CSE Biointelligence Lab, http://bi.snu.ac.kr/ 2
C t tC t tContentsContents
Theories of Language
Neuropsychology of Language and LanguageNeuropsychology of Language and Language
Neurophysiology of LanguageNeurophysiology of Language
(C) 2009, SNU CSE Biointelligence Lab, http://bi.snu.ac.kr/ 3
Th M l L i (1/4 )Th M l L i (1/4 )The Mental Lexicon (1/4 )The Mental Lexicon (1/4 )
Mental store of words and concepts
The characteristics of mental lexicon
♦ Semantic information: what is the word’s meaning?
♦ Syntactic information: how are words combined for sentences?
♦ Spelling: the details of word forms♦ Spelling: the details of word forms
♦ Pronunciation: sound pattern
♦ The representation of orthographic form: vision-based formsp g p
♦ The representation of phonological forms: sound-based forms
♦ There is a store of information about words in the brain
♦ No fixed contents: words can be forgotten and new words can be
learned, and more frequently used words are accessed more quickly.
♦ The neighborhood effect: accessing lexical (word) representations are
(C) 2009, SNU CSE Biointelligence Lab, http://bi.snu.ac.kr/ 5
♦ The neighborhood effect: accessing lexical (word) representations are
influenced by the so-called neighborhood effect.
Th M l L i (2/4 )Th M l L i (2/4 )The Mental Lexicon (2/4 )The Mental Lexicon (2/4 )
Th l l i i i d i f i ifiThe mental lexicon is organized as information-specific.
L l l d l l♦ Lexeme level: sound level
♦ Lemma level: grammatical
♦ Conceptual level: semantic
knowledge of words
O i d di tOrganized according to
Close connected: More
(C) 2009, SNU CSE Biointelligence Lab, http://bi.snu.ac.kr/ 6
Fig. 9.1: Fragment of a lexical network
according to the Levelt model
Th M l L i (3/4 )Th M l L i (3/4 )The Mental Lexicon (3/4 )The Mental Lexicon (3/4 )
S ti i i i l i l d i i t kSemantic priming in a lexical decision task
① Presented with pairs of wordsp
② The first one is a word, the second one (target) can be a real word or a
③ If the target is a real it can be related or unrelated in meaning to the prime③ If the target is a real, it can be related or unrelated in meaning to the prime.
④ The subject must decide whether the target is a word.
Faster and more accurate when target is preceded by a related prime (car-
truck) than an unrelated (tulip-truck).
Pronouncing the target, naming latencies are faster for related words thang g g
for unrelated words.
Semantic matching: active matching the meaning of the target with the
meaning of the prime
(C) 2009, SNU CSE Biointelligence Lab, http://bi.snu.ac.kr/ 7
meaning of the prime.
Th M l L i (4/4 )Th M l L i (4/4 )The Mental Lexicon (4/4 )The Mental Lexicon (4/4 )
Conceptual or semantic representation format
Words are represented by conceptual nodes and are connected
h theach other.
The strength of the connection and the distance between the
nodes are determined by the semantic relations or associativey
relations between the words.
(C) 2009, SNU CSE Biointelligence Lab, http://bi.snu.ac.kr/ 8
Fig. 9.2: An example of a semantic network. (Collins and Loftus, 1975)
Neural Substrates of the Mental LexiconNeural Substrates of the Mental LexiconNeural Substrates of the Mental LexiconNeural Substrates of the Mental Lexicon
and Conceptual Knowledgeand Conceptual Knowledge
Language deficits from various neurological problems –
understanding and producing the appropriate meaning of a word or
Wernicke’s aphasia: make errors in speech production (semantic
Semantic problems can be category-specific (e.g. living things vs. man-
There is a correlation between type of semantic deficits and area ofThere is a correlation between type of semantic deficits and area of
The semantic network is organized along lines of the conceptual
f d ( )categories of animacy and inanimacy (Caramazza).
(C) 2009, SNU CSE Biointelligence Lab, http://bi.snu.ac.kr/ 9
R l f B i I i S di (1/2 )R l f B i I i S di (1/2 )Results of Brain Imaging Studies (1/2 )Results of Brain Imaging Studies (1/2 )
f d h d f ffMRI studies support the idea of category-specific semantic
problem – separable neuronal circuits engaged
Fig. 9.3: Locations of
brain lesions that are
correlated withcorrelated with
selective deficits in
animals, or tools.
(C) 2009, SNU CSE Biointelligence Lab, http://bi.snu.ac.kr/ 10
R l f B i I i S di (2/2 )R l f B i I i S di (2/2 )Results of Brain Imaging Studies (2/2 )Results of Brain Imaging Studies (2/2 )
PET t d S b i ti t d h bj t dPET study: Same brain areas activated when subjects engaged
in naming persons (TP), animals (IT), and tools (IT+)
(C) 2009, SNU CSE Biointelligence Lab, http://bi.snu.ac.kr/ 11
Fig. 9.4: Activations in neurologically unimpaired subjects during naming of persons,
animals, or tools as determined by PET.
Th O i i f L i l L lTh O i i f L i l L lThe Organization of Lexical LevelThe Organization of Lexical Level
D i d h h fi di fl h i iDamasio proposed that the findings reflect the organization at
the word (lexical) level.
e.g. Patients having problems retrieving the name but itsg g p g
Fi 9 5 Th l l f t ti th t d d i h d ti ti
(C) 2009, SNU CSE Biointelligence Lab, http://bi.snu.ac.kr/ 12
Fig. 9.5: Three levels of representation that are needed in speech production: semantic
features, lexical nodes, and phonological segments.
P l A l f h Li i i IP l A l f h Li i i IPerceptual Analyses of the Linguistic InputPerceptual Analyses of the Linguistic Input
What enables understanding
of the linguistic input?g p
Differences in spoken input
analysis and written input
Fig. 9.6: Schematic representation of
the components that are involved in
spoken and written languagespoken and written language
comprehension. Notice that the flow
of information is bottom up, from
perceptual identification to “higher-
l l” d d l i i
(C) 2009, SNU CSE Biointelligence Lab, http://bi.snu.ac.kr/ 13
level” word and lemma activation.
S k I (1/2)S k I (1/2)Spoken Input (1/2)Spoken Input (1/2)
Phonemes: important building blocks of spoken language
Variability of the signal: speech sounds vary on the basis of the
context which they are spoken.y p
Lack of segmentation: phonemes often not separated – as little chunks
No clear silence between words in spoken sentencesNo clear silence between words in spoken sentences
Fi 9 8 S h f f th ti
(C) 2009, SNU CSE Biointelligence Lab, http://bi.snu.ac.kr/ 14
Fig. 9.7: Speech waveform for the word captain.
Fig. 9.8: Speech waveform for the question
“What do you mean?”
S k I (2/2)S k I (2/2)Spoken Input (2/2)Spoken Input (2/2)
Some researchers have proposed that sound representations are built
on spectral properties, others rejected that.
Fig. 9.9: Spectral
according to sounds.
The sound “ta” can beThe sound ta can be
analyzed to yield five
How do we break up the spoken input?
Through prosodic information: speech rhythm and voice pitch
(C) 2009, SNU CSE Biointelligence Lab, http://bi.snu.ac.kr/ 15
English: syllables carrying an accent or stress to establish word
N l S b f S k W d P iN l S b f S k W d P iNeural Substrates of Spoken Word ProcessingNeural Substrates of Spoken Word Processing
The superior temporal cortex – patients with bilateral lesions restricted to the
superior parts of the temporal lobe had specific difficulties in recognizing
speech sound: pure word deafness.
PET d fMRI t di H h l’ i d th i t l (STG)PET and fMRI studies: Heschel’s gyri and the superior temporal gyrus (STG)
of both hemispheres are activated by speech and nonspeech sounds alike – the
STG alone is not the seat of word comprehension.
The midsection of the superior temporal sulcus of both hemispheres (butThe midsection of the superior temporal sulcus, of both hemispheres (but
mostly of the left hemisphere) – plays an important role in speech
(C) 2009, SNU CSE Biointelligence Lab, http://bi.snu.ac.kr/ 16
Fig. 9.10: Superior temporal activations to speech and nonspeech sounds.
W i IW i I L R i iL R i iWritten InputWritten Input –– Letter RecognitionLetter Recognition
The pandemonium model: input is stored as iconic memory.
A computational model: 3 levels of representation –
Fig 9 12: Fragment of a connectionist
(C) 2009, SNU CSE Biointelligence Lab, http://bi.snu.ac.kr/ 17
Fig. 9.12: Fragment of a connectionist
network for letter recognition.
(McClelland & Rumelhart, 1981)
Fig. 9.11: The pandemonium model of
letter recognition of Selfridge (1959).
N l S b f W i W d P iN l S b f W i W d P iNeural Substrates of Written Word ProcessingNeural Substrates of Written Word Processing
The actual identification of
orthographic units may take place in
occipital-temporal regions of the leftp p g
Regions of the occipital-temporal
cortex activated preferentially incortex activated preferentially in
response to unpronounceable letter
Fig. 9.13: Activation of letters (L) compared to
activation of faces (F) in the ventral part of the left
(C) 2009, SNU CSE Biointelligence Lab, http://bi.snu.ac.kr/ 18
hemisphere (white circle in the brain scan).
h f dh f d llThe Recognition of WordsThe Recognition of Words –– Lexical AccessLexical Access
3 components of word processing:
lexical access, lexical selection,
From written text to word
representation: one or two
Dual-route reading models:
♦ From orthography to word formg p y
♦ By grapheme-to-phoneme
translation (indirect)( )
The cohort model of spoken word
(C) 2009, SNU CSE Biointelligence Lab, http://bi.snu.ac.kr/ 19
Fig. 9.14: Cohort model of spoken word recognition.
f d ( / )f d ( / )Brain Systems for Word Recognition (1/2)Brain Systems for Word Recognition (1/2)
Pathways for speech (auditory) processing:
♦ Heschl’s gyri of auditory cortex → Superior temporal gyrus →
Middle temporal g rus Inferior temporal g rus AngularMiddle temporal gyrus → Inferior temporal gyrus → Angular
gyrus (laterized more to the left hemisphere) → The left inferior
frontal gyrus, including the ventral part of Broca’s area for
Pathways for written input processing:
♦ Th i d d i l t f b th h i h♦ The primary and secondary visual cortex of both hemispheres →
Occipital-temporal regions of left hemisphere (identification of
orthographic units) → Middle temporal gyrus (phonographical
) f f f l l d lprocessing) → Left inferior frontal gyrus, including ventral part
of Broca’s area (translating orthographic input to phonological
information to pronounce
(C) 2009, SNU CSE Biointelligence Lab, http://bi.snu.ac.kr/ 20
B i S f W d R i i (2/2)B i S f W d R i i (2/2)Brain Systems for Word Recognition (2/2)Brain Systems for Word Recognition (2/2)
Fi 9 15 A hi hi l i t f h i
(C) 2009, SNU CSE Biointelligence Lab, http://bi.snu.ac.kr/ 21
Fig. 9.15: A hierarchical processing stream for speech processing.
f d ( / )f d ( / )Integration of Words in Sentences (1/3)Integration of Words in Sentences (1/3)
The role of context in word recognition – does linguistic
context influence word processing?
Contextual representations are crucial to determine aContextual representations are crucial to determine a
word’s sense and grammatical form.
Syntactic processing: integrating syntactic informationSyntactic processing: integrating syntactic information
For a full interpretation of linguistic input, we have to
i i l h iassign grammatical structure to the input.
Syntactic analysis possible even in the absence of real
Unlike words and syntactic properties, representations of
whole sentences are not stored in our brain.
(C) 2009, SNU CSE Biointelligence Lab, http://bi.snu.ac.kr/ 22
f d ( / )f d ( / )Integration of Words in Sentences (2/3)Integration of Words in Sentences (2/3)
The garden-path model (Frazier, 1987): processing in a quick way
Assumption: we minimizes what we have to do under the time
pressure of normal linguistic comprehension.p g p
♦ The minimal attachment mechanism: economic aspect
♦ The late closure mechanism: current processing focused
(C) 2009, SNU CSE Biointelligence Lab, http://bi.snu.ac.kr/ 23
Fig. 9.16: Constituent structure of a sentence.
(under the minimal attachment mechanism)
Fig. 9.17: Constituent structure of a sentence as
in Fig. 9.16. (nonminimal)
f d ( / )f d ( / )Integration of Words in Sentences (3/3)Integration of Words in Sentences (3/3)
Interactive models: our conceptual knowledge of order of events can
immediately influence the processing of the sentence, whereas
modular models would predict that this influence does not occur
l l h l l huntil later. The results are more in line with an interactive view.
Fig 9 18: ERPsFig. 9.18: ERPs
recorded in response to
sentences that start with
before (dashed line) or
(C) 2009, SNU CSE Biointelligence Lab, http://bi.snu.ac.kr/ 24
after (solid line).
l b fl b fNeural Substrates of Syntactic ProcessingNeural Substrates of Syntactic Processing
Agrammatic aphasia: caused by lesions in Broca’s area of the left
PET studies: the activation in Broca’s area increased reading
l t ti t tcomplex syntactic structure.
Syntactic processing increased activities in the anterior portions of
the superior temporal gyrus (STG), in the vicinity of area 22
Fig. 9.20: (a) PET activations in the anterior
portion of the superior temporal gyrus
related to syntactic processing. (b) Summary
f l i i h i i l
(C) 2009, SNU CSE Biointelligence Lab, http://bi.snu.ac.kr/ 25
of lesions in the anterior superior temporal
S h P d iS h P d iSpeech ProductionSpeech Production
Levelt ‘s modular model
for speech production
Brain damage can affect
each of the processing
Anomic patients with
extreme TOT (tip of theextreme TOT (tip of the
tongue): lexeme level
problem, not articulationp
Fi 9 21 O tli f th th f h
(C) 2009, SNU CSE Biointelligence Lab, http://bi.snu.ac.kr/ 26
Fig. 9.21: Outline of the theory of speech
production developed by Willem Levelt.
S h P d i (1/2)S h P d i (1/2)Speech Production (1/2)Speech Production (1/2)
Interactive view suggestion:
phonological activation begins
shortly after the semantic and
f fsyntactic information of
words has been activated, and
overlap in time.
ERP Test whether lemma (wor
d) selection proceeds
activation of the appropriate
l ( d)lexeme (sound), or
phonological information can
feed back and change the
ti ti l l f th lactivation level of the lemma
Fi 9 22 E l f ti li th t d i
(C) 2009, SNU CSE Biointelligence Lab, http://bi.snu.ac.kr/ 27
Fig. 9.22: Examples of stimuli that were used in an
ERP study of word production in Dutch.
S h P d i (2/2)S h P d i (2/2)Speech Production (2/2)Speech Production (2/2)
The LRP (lateralized readiness
potential): brain wave starts to
appear when a person plans
k b fto make a movement, before
the actual movement is
The correct LRP started to
develop even when the
subjects were not supposed to
Indication: lemma selection
might occur before the
phonological information at
the lexeme level is activated.
Fig. 9.23: Lateralized readiness potential response
(C) 2009, SNU CSE Biointelligence Lab, http://bi.snu.ac.kr/ 28
to go trials (blue line) and no-go trials (red line).
N l S b f S h P d iN l S b f S h P d iNeural Substrates of Speech ProductionNeural Substrates of Speech Production
Word generation activation – in the basal temporal
regions of the left hemisphere and in the left frontal
operculum (Broca’s area)operculum (Broca s area)
The articulation of words activation – in bilateral motor
cortex, the supplementary motor area (SMA), and thepp y ( )
Epileptic patients’ temporary inability to produce words
cortical stimulation of the basal temporal region of the– cortical stimulation of the basal temporal region of the
The activation in the frontal operculum might beThe activation in the frontal operculum might be
specific to phonological encoding in speech production.
A lesion in insula leads to apraxia of speech (a difficulty
d ) h ’ h
(C) 2009, SNU CSE Biointelligence Lab, http://bi.snu.ac.kr/ 29
in pronouncing words) in patients with Broca’s aphasia.
Neuropsychology of LanguageNeuropsychology of LanguageNeuropsychology of LanguageNeuropsychology of Language
and Language Disordersand Language Disorders
Hi i l Fi di i A h iHi i l Fi di i A h iHistorical Findings in AphasiaHistorical Findings in Aphasia
A h i ll i d fi i i l h i dAphasia: collective deficits in language comprehension and
production that accompany neurological damage.
Broca’s area: expressive aphasia – lesion in the posterior
portion of the left inferior frontal gyrus.
Wernicke’s area: receptive aphasia – fluent speech but speak
nonsensical sounds, words, and sentences.
(C) 2009, SNU CSE Biointelligence Lab, http://bi.snu.ac.kr/ 31
Fig. 9.24 (b): Broca’s area (in red). Fig. 9.25: Lateral view of Wernicke’s area.
Cl i l L li i i M d lCl i l L li i i M d lClassical Localizationist ModelClassical Localizationist Model
Lichtheim’s (1885) classic
model of language
Sometimes referred to as
the connectionist modelthe connectionist model
– not the McClelland &
Fig 9 26: Lichtheim’s (1885) classic model
(C) 2009, SNU CSE Biointelligence Lab, http://bi.snu.ac.kr/ 32
Fig. 9.26: Lichtheim s (1885) classic model
of language processing.
Cl ifi i f A h i (1/2 )Cl ifi i f A h i (1/2 )Classification of Aphasia (1/2 )Classification of Aphasia (1/2 )
Th 3 f lThe 3 parameters for language
♦ Spontaneous speech♦ Spontaneous speech
♦ Auditory comprehension
♦ Verbal repetition
Classifying Broca’s aphasia
♦ Single utterance pattern speech
♦ Sometimes from speech deficits –♦ Sometimes from speech deficits –
dysarthria (articulatory muscles
control loss), speech apraxia (deficits
to program articulation)to program articulation)
♦ limited to language production
(especially spoken language) and may
include comprehension deficits (e g
(C) 2009, SNU CSE Biointelligence Lab, http://bi.snu.ac.kr/ 33
include comprehension deficits (e.g.
agrammatism) Fig. 9.27: Speech problems of Broca’s
Cl ifi i f A h i (2/2 )Cl ifi i f A h i (2/2 )Classification of Aphasia (2/2 )Classification of Aphasia (2/2 )
Wernicke’s aphasia –
disorder of language
spoken or writtenspoken or written
language or both
Cannot speak meaningfulCannot speak meaningful
sentences even though
The posterior third of the
Fi 9 28 W i k ’ h i k fl tl
(C) 2009, SNU CSE Biointelligence Lab, http://bi.snu.ac.kr/ 34
Fig. 9.28: Wernicke’s aphasics can speak fluently.
Damage to Connections betweenDamage to Connections betweengg
Language AreasLanguage Areas
Conduction aphasia: A disconnection syndrome
occurring when the pathway from Wernickes to
B ’ i d dBroca’s area, is damaged.
Problems producing spontaneous speech as well as
repeating speech, and sometimes they use wordsepeat g speec , a d so et es t ey use o ds
Damage between conceptual representation areas can
harm the ability to comprehend spoken input but notharm the ability to comprehend spoken input but not
the ability to what was heard.
Lesions in the supramarginal and angular gyri regions
l f bilit t h d k i t– loss of ability to comprehend spoken input -- may
come from losing the ability to access semantic
information without losing syntactic or phonological
(C) 2009, SNU CSE Biointelligence Lab, http://bi.snu.ac.kr/ 35
Mechanisms of Aphasic DeficitsMechanisms of Aphasic Deficits
Comprehension deficits: from losses of stored linguistic
information? or from disruption of computational language
processes on the representations of linguistic inputs?processes on the representations of linguistic inputs?
Broca’s aphasia: agrammatic comprehension results from a
processing deficit; patients still have syntactic knowledge but they
cannot usecannot use.
Wernicke’s aphasia: when lexical-semantic processing is tested in
an implicit test (such as in the lexical decision task), patients do
not show impairment -- lexical-semantic knowledge might benot show impairment -- lexical-semantic knowledge might be
preserved but aphasics cannot access or exploit it
Authors’ view: patients’ brain cannot keep up with the time
challenges of language comprehension or production due tochallenges of language comprehension or production – due to
slow operation speed of accessing and exploiting stored linguistic
(C) 2009, SNU CSE Biointelligence Lab, http://bi.snu.ac.kr/ 36
Neurophysiology of LanguageNeurophysiology of Language
Functional Neuroimaging of LanguageFunctional Neuroimaging of Language
Metabolic correlates of aphasia
Focal brain lesions from stroke – widespread changesp g
in metabolism, extending to regions outside the
Fig. 9.29: (a)
Diagram of lesions
and (b) PET scan
showing regions of
in the brain of a
(C) 2009, SNU CSE Biointelligence Lab, http://bi.snu.ac.kr/ 38
in the brain of a
Electrophysiology of Language (1/2)Electrophysiology of Language (1/2)
The N400: a brain wave related to linguistic processes.
Increased when semantically mismatched
Fig. 9.30: ERP
words at the end of
d l land anomalous last
words that do not fit
specifications of the
(C) 2009, SNU CSE Biointelligence Lab, http://bi.snu.ac.kr/ 39
specifications of the
Electrophysiology of Language (2/2)Electrophysiology of Language (2/2)
Fig. 9.31: The
(C) 2009, SNU CSE Biointelligence Lab, http://bi.snu.ac.kr/ 40
S i P i d ERP (1/2)S i P i d ERP (1/2)Syntactic Processing and ERP (1/2)Syntactic Processing and ERP (1/2)
P600/SPS (the syntactic
positive shift): large
elicited by words after a
syntactic violationsyntactic violation
Fig. 9.32: ERPs from frontal(Fz),
l( ) d i l( ) lcentral(Cz), and parietal(Pz) scalp
recording sites elicited in response to
each word of sentences that are
anomalous versus those that are
(C) 2009, SNU CSE Biointelligence Lab, http://bi.snu.ac.kr/ 41
anomalous versus those that are
S i P i d ERP (2/2)S i P i d ERP (2/2)Syntactic Processing and ERP (2/2)Syntactic Processing and ERP (2/2)
LAN (l ft t i ti it ) ti thLAN (left anterior negativity): negative wave over the
left frontal areas when words violates the required word
category in a sentence
e.g. “the red eats,” “he mow”
Fig. 9.33: ERPs
related to semantic
(C) 2009, SNU CSE Biointelligence Lab, http://bi.snu.ac.kr/ 42
Only human possess a true language.
Classical models are not enough – detailed analysisClassical models are not enough – detailed analysis
of neurological lesions, functional neuroimaging,
human electrophysiology and computationalhuman electrophysiology, and computational
modeling can provide surprising modification.
Th h l t i lThe human language system is complex.
Promising if psycholinguistic models combine with
(C) 2009, SNU CSE Biointelligence Lab, http://bi.snu.ac.kr/ 43
Th ht Q tiTh ht Q tiThought QuestionsThought Questions
1. How might the mental lexicon be organized in the brain?
Would one expect to find it localized in a particular spot in
cortex? If not, why not?, y
2. At what stage of input processing are the comprehension of
spoken and of written language the same, and where must they
be different? Are there any exceptions to this rule?be different? Are there any exceptions to this rule?
3. Describe the route that an auditory speech signal might take in
the cortex, from perceptual analysis to comprehension.
4 What evidence exists for the role of the right hemisphere in4. What evidence exists for the role of the right hemisphere in
language processing? If the right hemisphere has a role in
language, what might that be?
5 Wh t l if d ki l i h i ?5. What role, if any, does working memory play in comprehension?
Give an example of how disruptions in working memory might
affect the understanding of spoken or written inputs.
(C) 2009, SNU CSE Biointelligence Lab, http://bi.snu.ac.kr/ 45