Robust Multimodal Cognitive Load Measurement (eBook)

Preface 6
Acknowledgements 8
Contents 10
Part I: Preliminaries 16
Chapter 1: Introduction 17
1.1 What Is Cognitive Load 18
1.2 Background 19
1.3 Multimodal Cognitive Load Measurement 20
1.4 Structure of the Book 22
References 26
Chapter 2: The State-of-The-Art 27
2.1 Working Memory and Cognitive Load 27
2.2 Subjective Measures 29
2.3 Performance Measures 30
2.4 Physiological Measures 32
2.5 Behavioral Measures 33
2.6 Estimating Load from Interactive Behavior 37
2.7 Measuring Different Types of Cognitive Load 38
2.8 Differences in Cognitive Load 39
2.8.1 Gender Differences in Cognitive Load 39
2.8.2 Age Differences in Cognitive Load 39
2.8.3 Static Graphics Versus Animated Graphics in Cognitive Load 40
2.9 Summary 41
References 41
Chapter 3: Theoretical Aspects of Multimodal Cognitive Load Measures 47
3.1 Load? What Load? Mental? Or Cognitive? Why Not Effort? 48
3.2 Mental Load in Human Performance 48
3.2.1 Mental Workload: The Early Years 49
3.2.2 Subjective Mental Workload Scales and Curve 52
3.2.3 Cognitive Workload and Physical Workload Redlines 53
3.3 Cognitive Load in Human Learning 54
3.3.1 Three Stages of CLT: The Additivity Hypothesis 56
3.3.2 Schema Acquisition and First-in Method 57
3.3.3 Modality Principle in CTML 58
3.3.4 Has Measuring Cognitive Load Been a Means to Advancing Theory? 59
3.3.5 Bridging Mental Workload and Cognitive Load Constructs 63
3.3.6 CLT Continues to Evolve 64
3.4 Multimodal Interaction and Cognitive Load 65
3.4.1 Multimodal Interaction and Robustness 65
3.4.2 Cognitive Load in Human Centred Design 69
3.4.3 Dual Task Methodology for Inducing Load 69
3.4.4 Workload Measurement in a Test and Evaluation Environment 70
3.4.5 Working Memory´s Workload Capacity: Limited But Not Fixed 72
3.4.6 Load Effort Homeostasis (LEH) and Interpreting Cognitive Load 73
3.5 Multimodal Cognitive Load Measures (MCLM) 77
3.5.1 Framework for MCLM 77
3.5.2 MCLM and Cognitive Modelling 79
3.5.3 MCLM and Decision Making 79
3.5.4 MCLM and Trust Studies 80
3.6 Summary 80
References 81
Part II: Physiological Measurement 86
Chapter 4: Eye-Based Measures 87
4.1 Pupillary Response for Cognitive Load Measurement 87
4.2 Cognitive Load Measurement Under Luminance Changes 89
4.2.1 Task Design 89
4.2.2 Participants and Apparatus 90
4.2.3 Subjective Ratings 90
4.3 Pupillary Response Features 91
4.4 Workload Classification 92
4.4.1 Feature Generation for Workload Classification 93
4.4.2 Feature Selection and Workload Classification 94
4.4.3 Results on Pupillary Response 96
4.5 Summary 96
References 97
Chapter 5: Galvanic Skin Response-Based Measures 98
5.1 Galvanic Skin Response for Cognitive Load Measurement 98
5.2 Cognitive Load Measurement in Arithmetic Tasks 99
5.2.1 Task Design 99
5.2.2 GSR Feature Extraction 100
5.2.2.1 Time Domain Features 100
5.2.2.2 Frequency Domain Features 101
5.2.3 Feature Analyses 102
5.3 Cognitive Load Measurement in Reading Tasks 104
5.3.1 Task Design 104
5.3.2 GSR Feature Extraction 105
5.3.3 Feature Analyses 105
5.4 Cognitive Load Classification in Arithmetic Tasks 106
5.4.1 Features for Workload Classification 106
5.4.2 Classification Results 107
5.5 Summary 108
References 109
Part III: Behavioural Measurement 111
Chapter 6: Linguistic Feature-Based Measures 112
6.1 Linguistics 112
6.2 Cognitive Load Measurement With Non-Word Linguistics 113
6.3 Cognitive Load Measurement with Words 115
6.3.1 Word Count and Words per Sentence 115
6.3.2 Long Words 115
6.3.3 Positive and Negative Emotion Words 115
6.3.4 Swear Words 116
6.3.5 Cognitive Words 116
6.3.6 Perceptual Words 116
6.3.7 Inclusive Words 116
6.3.8 Achievement Words 117
6.3.9 Agreement and Disagreement Words 117
6.3.10 Certainty and Uncertainty Words 117
6.3.11 Summary of Measurements 117
6.4 Cognitive Load Measurement Based on Personal Pronouns 119
6.5 Language Complexity as Indices of Cognitive Load 120
6.5.1 Lexical Density 120
6.5.2 Complex Word Ratio 120
6.5.3 Gunning Fog Index 121
6.5.4 Flesch-Kincaid Grade 121
6.5.5 SMOG Grade 121
6.5.6 Summary of Language Measurements 122
6.6 Summary 122
References 123
Chapter 7: Speech Signal Based Measures 124
7.1 Basics of Speech 125
7.2 Cognitive Load Experiments 125
7.2.1 Reading Comprehension Experiment 125
7.2.2 Stroop Test 127
7.2.3 Reading Span Experiment 127
7.2.4 Time Constraint 128
7.2.5 Experiment Validation 129
7.3 Speech Features and Cognitive Load 129
7.3.1 Source-Based Features 130
7.3.2 Filter-Based Features 130
7.4 A Comparison of Features for Cognitive Load Classification 132
7.4.1 Pitch and Intensity Features 132
7.4.2 EGG Features 133
7.4.3 Glottal Flow Features 135
7.5 Cognitive Load Classification System via Speech 138
7.6 Summary 138
References 139
Chapter 8: Pen Input Based Measures 141
8.1 Writing Based Measures 141
8.2 Datasets for Writing-Based Cognitive Load Examination 143
8.2.1 CLTex Dataset 144
8.2.2 CLSkt Dataset 145
8.2.3 CLDgt Dataset 146
8.3 Stroke-, Substroke- and Point-Level Features 147
8.4 Cognitive Load Implications on Writing Shapes 149
8.5 Cognitive Load Classification System 151
8.6 Summary 152
References 153
Chapter 9: Mouse Based Measures 154
9.1 User Mouse Activity 154
9.2 Mouse Features for Cognitive Load Change Detection 155
9.2.1 Temporal Features 155
9.2.2 Spatial Features 158
9.2.2.1 Spatial Features with Straight Lines 159
9.2.2.2 Straight Line Results 161
9.3 Limitations of Mouse Feature Measurements 162
9.4 Mouse Interactivity in Multimodal Measures 163
9.5 Summary 163
References 164
Part IV: Multimodal Measures and Affecting Factors 165
Chapter 10: Multimodal Measures and Data Fusion 166
10.1 Multimodal Measurement of Cognitive Load 166
10.2 An Abstract Model for Multimodal Assessment 167
10.3 Basketball Skills Training 169
10.4 Subjective Ratings and Performance Results 170
10.5 Individual Modalities 172
10.6 Multimodal Fusion 174
10.7 Summary 176
References 176
Chapter 11: Emotion and Cognitive Load 177
11.1 Emotional Arousal and Physiological Response 177
11.2 Cognitive Load Measurement with Emotional Arousal 178
11.2.1 Task Design 178
11.2.2 Pupillary Response Based Measurement 179
11.2.3 Skin Response Based Measurement 180
11.3 Cognitive Load Classification with Emotional Arousal 181
11.3.1 Cognitive Load Classification Based on Pupillary Response 182
11.3.2 Cognitive Load Classification Based on GSR 183
11.3.3 Cognitive Load Classification Based on the Fusion 184
11.4 Summary 186
References 186
Chapter 12: Stress and Cognitive Load 188
12.1 Stress and Galvanic Skin Response 188
12.2 Cognitive Load Measurement Under Stress Conditions 189
12.2.1 Task Design 189
12.2.2 Procedures 190
12.2.3 Subjective Ratings 191
12.3 GSR Features Under Stress Conditions 191
12.3.1 Mean GSR Under Stress Conditions 191
12.3.2 Peak Features Under Stress Conditions 193
12.4 Summary 196
References 197
Chapter 13: Trust and Cognitive Load 198
13.1 Definition of Trust 198
13.2 Related Work 199
13.2.1 Trust 199
13.2.2 Trust and Cognitive Load 200
13.3 Trust of Information and Cognitive Load 202
13.3.1 Task Design 202
13.3.2 Data Collection 204
13.3.2.1 Survey Responses 205
13.3.2.2 Behavioral Measures 205
13.3.2.3 Performance Measures 205
13.4 Data Analyses 206
13.5 Analysis Results 207
13.5.1 Subjective Ratings of Mental Effort 207
13.5.2 Linguistic Analysis of Think-Aloud Speech 207
13.5.2.1 Pause Analysis 208
13.5.2.2 Linguistic Category Analysis 210
13.5.2.3 Other Behavioral Features 213
13.6 Interpersonal Trust and Cognitive Load 214
13.6.1 Task Design 214
13.6.2 Results 214
13.7 Summary 215
References 216
Part V: Making Cognitive Load Measurement Accessible 218
Chapter 14: Dynamic Cognitive Load Adjustments in a Feedback Loop 219
14.1 Dynamic Cognitive Load Adjustments 219
14.2 Dynamic Workload Adaptation Feedback Loop 220
14.2.1 Task Design 220
14.2.2 Procedures 221
14.3 GSR Features 222
14.3.1 Signal Processing 222
14.3.2 Feature Extraction 223
14.4 Cognitive Load Classification 224
14.4.1 Offline Cognitive Load Classifications 224
14.4.2 Online Cognitive Load Classifications 225
14.5 Dynamic Workload Adjustment 227
14.5.1 Adaptation Models 227
14.5.2 Performance Evaluation of Adaptation Models 228
14.6 Summary 229
References 229
Chapter 15: Real-Time Cognitive Load Measurement: Data Streaming Approach 230
15.1 Sliding Window Implementation 231
15.2 Streaming Mouse Activity Features 232
15.3 Lessons Learnt 233
15.4 Summary 235
References 235
Chapter 16: Applications of Cognitive Load Measurement 236
16.1 User Interface Design 236
16.2 Emergency Management 239
16.3 Driving and Piloting 241
16.4 Education and Training 242
16.5 Other Applications 244
16.6 Future Applications 245
References 246
Part VI: Conclusions 249
Chapter 17: Cognitive Load Measurement in Perspective 250
References 253