Multisensory Object Perception in the Primate Brain (eBook)

Foreword 5
Contents 7
Contributors 9
1 General Introduction 12
Part I Mechanisms 16
2 Corticocortical Connectivity Subserving Different Forms of Multisensory Convergence 17
2.1 Introduction 17
2.2 Bimodal Properties: The Superior Temporal Sulcus 17
2.2.1 Bimodal Properties: The Anterior Ectosylvian Sulcus 18
2.2.2 Bimodal Properties: The Superior Colliculus 20
2.3 Non-bimodal Forms of Multisensory Processing 21
2.3.1 Subthreshold Multisensory Processing and Crossmodal Cortical Connectivity 23
2.3.2 Crossmodal Cortical Connections and Multisensory Integration 23
2.4 Multisensory Convergence: A Continuum? 25
2.4.1 Cortical Diversity: Differential Distributions of Multisensory Neuron Types? 25
2.5 Conclusions 28
References 28
3 Computational Modeling of Multisensory Object Perception 31
3.1 Introduction 31
3.2 Empirical Evidence for Crossmodal Object Perception 32
3.3 Computational Principles Evident from the Experimental Data 33
3.3.1 Sensory Integration 34
3.3.2 Sensory Combination 34
3.3.3 Integration of Measurements with Prior Knowledge 35
3.3.4 Explaining Away 35
3.3.5 Using the Appropriate Model 36
3.3.6 Decision Making 36
3.3.7 Learning of Cue Reliabilities and Priors 37
3.3.8 Task Dependence 37
3.3.9 Developmental Learning 37
3.4 Models of Multimodal Object Perception 38
3.4.1 Ideal Observer Models of Multimodal Object Perception 39
3.4.2 Intermediate Models of Multimodal Object Perception 48
3.4.3 Neural Models Implementing Multimodal Perception 49
3.5 Open Questions 52
3.5.1 How Are Model Parameters Learned? 52
3.5.2 How Are the Likelihood Functions Learned? 53
3.5.3 Where Does the Prior Come From? 53
3.5.4 How Does the Brain Learn the Appropriate Generative Model, If at All? 54
3.5.5 How Are Different Models Combined? 56
3.5.6 Does the Brain Represent Full Probability Distributions or Implicit Measures of Uncertainties? 56
3.5.7 How Ideal Is Human Learning? 57
3.5.8 How Do Laboratory Experiments Transfer to Natural Tasks? 57
3.5.9 The Role of Approximations 58
3.6 Conclusion 58
References 59
4 Methodological Considerations: Electrophysiology of Multisensory Interactions in Humans 64
4.1 Introduction 64
4.1.1 Specificities of the Electrophysiological Techniques 64
4.2 The Additive Model in Human Electrophysiology 65
4.2.1 Potential Biases and Artifacts Generated by the Additive Model 66
4.2.1.1 Activities Common to All (Bimodal and Unimodal) Stimuli 66
4.2.1.2 Unimodal Deactivation of Sensory Cortices: Block vs Random Designs 67
4.2.1.3 More About Attentional Effects in the Additive Model 68
4.2.2 Suitability and Necessity of the Additive Model in Human Electrophysiology 69
4.2.3 Topographic Interpretation of : Nature of the Cross-Modal Interactions 71
4.3 The Additive Model in EEG/MEG, Hemodynamic, and Single-Neuron Data: One Model, Three Interpretations 72
4.3.1 Comparison with Hemodynamic Data 72
4.3.2 Comparison with Single-Unit Data 73
4.4 Statistical Assessment of Cross-Modal Interactions 74
4.4.1 At a Group Level 74
4.4.2 At an Individual Subject Level 74
4.4.3 Correction for Multiple Testing 76
4.5 Conclusion 77
References 77
5 Cortical Oscillations and Multisensory Interactions in Humans 80
5.1 Introduction 80
5.2 Topographically Unspecific or Widespread Activity 81
5.3 Localized Activity 82
5.4 Cortico-cortical Interactions 86
5.5 Conclusions 87
References 88
6 Multisensory Functional Magnetic Resonance Imaging 92
6.1 Introduction 92
6.2 Limitations and Strengths of fMRI 93
6.3 Detection of Multisensory Integration Regions 94
6.4 Connectivity 96
6.4.1 Anatomical Connectivity 97
6.4.2 Functional Connectivity 97
6.4.3 Effective Connectivity 97
6.4.4 Representational Connectivity 98
6.5 Conclusion and Outlook 98
References 99
Part II Audio-Visual Integration 102
7 Audiovisual Temporal Integration for Complex Speech, Object-Action, Animal Call, and Musical Stimuli 103
7.1 Introduction 103
7.2 Multisensory Integration and Temporal Synchrony 104
7.3 The Mechanisms Underlying Multisensory Temporal Perception 106
7.4 Measuring Temporal Perception 107
7.5 Audiovisual Temporal Perception for Simple Stimuli 108
7.6 Audiovisual Temporal Perception for Complex Stimuli 110
7.7 Factors Affecting the Audiovisual Temporal Perception of Complex Speech and Nonspeech Stimuli 115
7.7.1 How Does Stimulus Type Affect Audiovisual Temporal Perception? 116
7.7.2 How Do the Physical Characteristics in the Articulation of a Speech Stimulus Affect Audiovisual Temporal Perception? 118
7.7.3 Does the Visual Orientation of a Stimulus Affect Audiovisual Temporal Perception? 120
7.7.4 What Role Does the 'Unity Effect' Play in Audiovisual Temporal Perception? 120
7.8 Conclusions 123
References 124
8 Imaging Cross-Modal Influences in Auditory Cortex 130
8.1 Introduction 130
8.2 Cross-Modal Influences in Auditory Cortex Revealed by fMRI 131
8.3 A Functional Parcellation of Auditory Cortex 134
8.4 Localizing Cross-Modal Influences to Individual Fields 136
8.5 Extrapolation from Functional Imaging to Neuronal Responses 139
8.6 Conclusions 140
References 140
9 The Default Mode of Primate Vocal Communication and Its Neural Correlates 145
9.1 Introduction 145
9.2 Faces and Voices Are Inextricably Linked in Primates 145
9.3 Neocortical Bases for Integrating Faces and Voices 148
9.4 Auditory Cortical Interactions with the Superior Temporal Sulcus Mediates Face/Voice Integration 150
9.5 Beyond Visual Influences in Auditory Cortex 152
9.6 The Development of Multisensory Systems for Communication 153
9.7 Conclusion 155
References 155
10 Audio-Visual Perception of Everyday Natural Objects Hemodynamic Studies in Humans 160
10.1 Introduction 160
10.2 Overview of Audio-Visual Interaction Sites in Human Cortex 162
10.3 Background of System-Level Mechanisms for Multisensory Integration 167
10.4 Meta-analyses of Brain Networks Involved in Audio-Visual Interactions 170
10.4.1 Networks That Reflect Intermodal Invariant Audio-Visual Attributes 170
10.4.2 Networks for Audio-Visual Integration of Natural Objects and Faces in Motion 174
10.4.3 Networks for Audio-Visual Integration Using Static Pictures of Objects 177
10.4.4 Interactions When Using Artificial or Abstract Audio-Visual Pairings 178
10.4.5 Networks for Semantically Mis-matched Auditory and Visual Pairs 180
10.5 Category-Specific Audio-Visual Object Processing and Knowledge Representations 185
10.6 Conclusions 186
References 187
11 Single-Trial Multisensory Learning and Memory Retrieval 196
11.1 Background 196
11.2 Findings 198
11.2.1 Multisensory Experiences Impact Subsequent Visual Memory Performance 198
11.2.2 Effects on Memory Performance Are Dissociable from Encoding 201
11.2.3 The Role of Attention, Alerting, and Novelty 202
11.2.4 Visual Stimuli Are Rapidly Discriminated Within Lateral Occipital Cortices According to Past Multisensory Experiences 203
11.3 Implications 204
11.4 Conclusions and Future Directions 209
References 210
Part III Visuo-Tactile Integration 214
12 Multisensory Texture Perception 215
12.1 Introduction 215
12.2 Texture and Its Measurement 216
12.3 Haptic Roughness Perception 218
12.4 Visual and Visual/Haptic Texture Perception 222
12.5 Auditory Texture Perception 226
12.6 Brain Correlates of Texture Perception 229
12.7 Final Comments 230
References 231
13 Dorsal and Ventral Cortical Pathways for Visuo-haptic Shape Integration Revealed Using fMRI 235
13.1 Introduction 235
13.2 Visual Cortical Pathways for Action and Perception 236
13.3 Converging Visual and Somatosensory Pathways 237
13.4 Measuring Neuronal Convergence with BOLD fMRI 241
13.5 Sites of Visuo-haptic Neuronal Convergence 244
13.6 Conclusions 250
References 250
14 Visuo-haptic Perception of Objects and Scenes 255
14.1 Introduction 255
14.2 Evidence for Common Principles of Functional Organisation Across Vision and Touch 256
14.3 Evidence for Common Principles of Information Processing Across Vision and Touch for Object Recognition 260
14.3.1 Shape Constancy and the Visual Recognition of Objects Across Changes in Viewpoint 260
14.3.2 Shape Constancy and the Recognition of Static Objects in Vision and Touch 263
14.3.3 Shape Constancy and the Recognition of Dynamic Objects in Vision and Touch 265
14.4 Evidence for Common Information Processing of Where Information Across Vision and Touch 267
14.4.1 Visual and Haptic Spatial Updating of Scenes 267
14.4.2 Crossmodal Updating in Scene Perception 269
14.4.3 The Role of Noninformative Visual Information on Haptic Scene Perception 270
14.5 Conclusions and Future Directions 271
References 273
15 Haptic Face Processing and Its Relation to Vision 276
15.1 Overview 276
15.2 Facial Identity 277
15.2.1 Visual Perception of Facial Identity 277
15.2.1.1 How Does the Visual System Process Facial Identity? 277
15.2.1.2 How Does the Visual System R epresent Facial Identity? 279
15.2.1.3 What Are the Neural Mechan isms that Underlie Visual Perception of Facial Identity? 279
15.2.2 Haptic Perception of Facial Identity 281
15.2.2.1 How Does the Haptic System Process Facial Identity and How Does This Relate to Vision? 281
15.2.2.2 How Does the Haptic System Represent Facial Identity and How Does This Relate to Vision? 283
15.2.2.3 What Are the Neural Mechanisms that Underlie Haptic Perception of Facial Identity and How Does This Relate to Vision? 284
15.2.3 Summary 287
15.3 Facial Expressions of Emotion 288
15.3.1 Visual Perception of Emotion from Facial Expressions 288
15.3.1.1 How Does the Visual System Process Facial Expressions of Emotion? 288
15.3.1.2 How Does the Visual System Represen t Facial Expressions of Emotion? 289
15.3.1.3 What Are the Neural Mechanisms that Underlie Visual Perception of Facial Expressions of Emotion? 290
15.3.2 Haptic Processing of Emotion from Facial Expressions 291
15.3.2.1 How Does the Haptic System Process Facial Expressions of Emotion and How Does This Relate to Vision? 291
15.3.2.2 How Does the Haptic System Represent Facial Expressions of Emotion and How Does This Relate to Vision? 294
15.3.2.3 What Are the Neural Mechanisms that Underlie Haptic Perception of Facial Expressions of Emotion and How Does This Relate to Vision? 295
15.3.3 Summary 296
15.4 General Summary, Conclusions, and Future Directions 296
15.4.1 How Are Faces Processed? 296
15.4.2 How Are Faces Represented? 297
15.4.3 What Are the Underlying Neural Mechanisms and How Does this Relate to Vision? 297
References 298
Part IV Plasticity 304
16 The Ontogeny of Human Multisensory Object Perception: A Constructivist Account 305
16.1 Introduction 305
16.2 Setting the Theoretical Problem 306
16.3 Response to A-V Intensity Relations 308
16.4 Response to A-V Temporal Synchrony Relations 309
16.4.1 Infant Perception of A-V Synchrony Relations 310
16.4.2 Infant Perception of A-V Speech Synchrony and Effects of Experience 311
16.4.3 Binding of Multisensory Attributes 313
16.4.4 Binding of Nonnative Faces and Vocalizations 314
16.4.5 The Importance of Spatiotemporal Coherence in Object Perception 318
16.4.6 Summary of Effects of A-V Temporal Synchrony on Infant Perception 319
16.5 Response to A-V Colocation 320
16.6 Perception of Multisensory Sequences in Infancy 322
16.7 Conclusion 326
References 326
17 Neural Development and Plasticity of MultisensoryRepresentations 330
17.1 A Brief Introduction to Multisensory Processes 330
17.2 Neural Processing in Adult Multisensory Circuits 331
17.3 The Development of Subcortical Multisensory Representations 332
17.4 The Development of Cortical Multisensory Representations 335
17.5 The Maturation of the Multisensory Integrative Principles 336
17.6 Sensory Experience as a Key Determinant in Multisensory Development 337
17.7 The Mechanistic Bases for Multisensory Development? 340
17.8 Spatial Receptive Fields and Spatiotemporal Receptive Fields: New Tools to Evaluate Multisensory Representations and Their Development 341
17.9 Multisensory Studies in Awake and Behaving Preparations 344
17.10 Moving Toward the Development and Plasticity of Multisensory Object Representations 345
References 346
18 Large-Scale Brain Plasticity Following Blindness and the Use of Sensory Substitution Devices 351
18.1 Introduction 351
18.2 The Plastic Brain 353
18.2.1 Plasticity Across the Lifespan 354
18.3 Plastic Changes Following Sensory Loss 355
18.4 Developmental and Adult Plasticity Following Blindness 361
18.5 Rehabilitation in the Case of Blindness and Severe Visual Impairment 363
18.5.1 Sensory Substitution Devices 363
18.5.2 Sensory Restoration Approaches 369
18.6 Concluding Remarks 372
References 373
Index 381