Multimedia Interaction and Intelligent User Interfaces (eBook)

Preface 4
Contents 7
Retrieving Human Actions Using Spatio-Temporal Features and Relevance Feedback 9
Introduction 9
Action Retrieval Scheme 12
Action Retrieval Framework 12
Spatio-Temporal Interest Point Detection 13
Feature Description 14
Codebook Formation and Action Video Representation 16
Similarity Matching Scheme 16
Action Retrieval on the KTH Dataset 16
Dataset Processing 16
Performance Evaluation 17
Summary for Experiments on the KTH Dataset 21
Realistic Action Retrieval in Movies 22
Challenges of This Task 22
Implementation 24
Result Demonstration 26
Discussion 28
Application 29
Conclusions 29
References 30
Computationally Efficient Clustering of Audio-Visual Meeting Data 32
Introduction 32
Background 33
Challenges in Meeting Analysis 35
Background on Speaker Diarization 37
Background on Audio-Visual Synchrony 38
Human Body Motions in Conversations 39
Approach 40
The Augmented MultiParty Interaction (AMI) Corpus 41
Audio Speaker Diarization 43
Traditional Offline Speaker Diarization 43
Feature Extraction 43
Speech/Nonspeech Detection 43
Speaker Segmentation and Clustering 44
Online Speaker Diarization 45
Unsupervised Bootstrapping of Speaker Models 45
Speaker Recognition 46
A Note on Model Order Selection 46
Summary of the Diarization Performance 47
Extracting Computationally Efficient Video Features 48
Estimating Personal Activity Levels in the Compressed Domain 49
Finding Personal Head and Hand Activity Levels 50
Estimating Speakers Using Video Only 53
Associating Speaker Clusters with Video Channels 55
Audio-Visual Clustering Results 57
Using Raw Visual Activity 57
Using Estimates of Speaking Activity from Video 58
Discussion 60
References 62
Cognitive-Aware Modality Allocation in Intelligent Multimodal Information Presentation 67
Introduction 67
Modality and Human Information Processing 69
Modality and Sensory Processing 70
Modality and Perception 71
Visual Attention 71
Auditory Attention 71
Cross-Modal Attention 72
Modality and Working Memory 72
Working Memory Theory 73
Dual Coding Theory 73
Relating the Two Theories 74
Experiment on Modality Effects in High-Load HCI 75
Presentation Material 76
Task and Procedure 77
Measurements 77
Hypotheses 78
Results on Performance, Cognitive Load and Stress 78
Performance 78
Cognitive Load and Stress 80
Discussion 81
Text vs. Image 81
Visual Aid vs. Auditory Aid 81
Verbal Aid vs. Nonverbal Aid 82
Additional Aid vs. No Aid 83
Low Load vs. High Load 83
A Modality Suitability Prediction Model 84
Conclusions 86
References 86
Natural Human-Computer Interaction 90
Introduction 90
From Ergonomics to Human-Computer Interaction 90
Multimodal Interfaces 91
Natural Human-Computer Interaction 92
Natural Interaction Systems 92
Human-Centered Design 93
Intuitive Interaction 93
Natural Language and Tangible User Interfaces 94
Sensing Human Behavior 95
Sensed Spaces and Sensors Categories 95
Optical Sensors and Computer Vision Technologies 96
Image Analysis Techniques 96
Tracking Techniques 96
Observing Human Activity 96
People Detection 97
People Tracking 97
Gaze Estimation 98
State of the Art 99
Interactive Tabletop 99
Tangible User Interface 100
Smart Room 101
Smart Room with Tangible Natural Interaction 102
TANGerINE Smart Room: a Case Study 102
TANGerINE Smart Cube 104
Manipulation State Awareness 105
Gesture Detection Algorithm 105
Bluetooth-Based Proximity Awareness 105
Computer Vision Applied to the TANGerINE Platform 106
Observing Human Activity in TANGerINE Smart Room 107
References 108
Gesture Control for Consumer Electronics 112
Introduction 112
Sensing Technologies 113
Haptics 114
Handhold Sensors 114
Vision 114
Ultrasound 114
Infrared Proximity Sensing 115
Vision-Based Gesture Recognition 115
Body Part Detection 117
Gesture Tracking 119
Gesture Recognition 122
Gesture Control: Products and Applications 125
GestureTek 125
Toshiba 125
Mgestyk 126
Fraunhofer 126
TVs or Displays 126
Gaming 127
Mobile Phones 127
Automobiles 127
Conclusions 128
References 129
Empirical Study of a Complete System for Real-Time Face Pose Estimation 134
Introduction 134
Problem Definition 136
Problem Statement 136
Pose Estimation Algorithm 137
3D Mesh 138
Texture Extraction 138
Automatic Initialization 138
Face and Feature Detection 138
Mesh Initialization 140
Tracking 141
Overview of Method 141
2D Feature Tracking 142
Adaptation Step 142
Matching Criterion 143
Detection on Mesh Texture 145
Reconstruction Error of Mesh Texture 145
Detection of Failed Tracking 149
Results 150
Stability Analysis for Static Images 150
Accuracy on Videos 152
Semi-Automatic Annotation 152
Performance of Different Search Strategies: Angular Error 154
Performance of Different Search Strategies: MSE 155
Performance of Different Search Strategies: Computation Time 155
Influence of Texture Representation 156
Influence of Training Size 156
Benefits of the Proposed System 160
Analysis of Typical Results 162
Examples of Tracking Failure 164
Conclusions 165
References 165
Evolution-based Virtual Content Insertion with Visually Virtual Interactions in Videos 168
Introduction 168
System Overview 170
Essential Ideas 170
System Overview 171
Video Content Analysis 172
Frame Profiling 172
Motion Estimation 173
Region Segmentation 173
ROI Estimation 174
Aural Saliency Analysis 174
Virtual Content Analysis 175
Virtual Content Characterization 176
Behavior Modeling 178
The Cell Phase 179
The Microbe Phase 179
The Creature Phase 180
Virtual Content Insertion 182
Animation Generation 182
Layer Composition 184
Experimental Results 185
Summary 188
References 189
Physical Activity Recognition with Mobile Phones: Challenges, Methods, and Applications 190
Introduction 191
Background of Physical Activity Recognition 191
Practical Challenges on Mobile Devices 193
Accelerometer Based Physical Activity Recognition Methods 194
Data Format 194
Accelerometer Sensor Calibration 196
Signal Projection 200
Data Collection 201
Feature Extraction and Selection 202
Classification Algorithms 204
Smoothing Algorithms 206
System Design and Implementation 207
Applications and Use Cases 209
Physical Activity Diary 209
Mobile Healthcare and Wellness 212
Human-Centric Sensing and Sharing in Mobile Social Networks 213
User Interfaces 213
Conclusion and Future Work 215
References 217
Gestures in an Intelligent User Interface 219
Two Sides of the Same Coin 219
Related Work 221
A Human's Perspective 221
A System's Perspective 222
Experiment 1: Intuitive Gesturing 223
Method 224
Setup 224
Results 226
Condition Qx 226
Condition Xp 227
Sample Summary 227
Commands: Pointing 228
Commands: Selecting 229
Commands: Deselecting 230
Commands: Resizing 231
Conclusion 233
Experiment 2: Gesturing in the Interface 234
Method 234
Out-of-Range and Tracking 236
Select and Deselect 236
Rotate 236
Resizing 236
Restore and Remove 237
Results 237
Questionnaire 238
Observations 240
Conclusion 240
Conclusion and Discussion 242
References 243
Video Summary Quality Evaluation Based on 4C Assessment and User Interaction 247
Introduction 247
Related Work 249
Uniform Framework for Video Summary Quality Evaluation 251
Summary Unit Sequence Generation 252
Frame Alignment-Based Summary Unit Matching 252
Similarity-Based Automatic 4C Assessment 255
Coverage Assessment 255
Conciseness Assessment 257
Coherence Assessment 257
Context Assessment 259
User Interaction Based Individual Evaluation 260
User Interaction Based Requirement Gathering 261
Transformation of 4C Assessment Scores 261
Incremental User Interaction 264
Experiments 264
Validation of 4C Assessment Algorithm 265
Validation of Incremental User Interaction 269
Validation of Evaluation Result Transformation 270
Conclusions 271
References 271
Multimedia Experience on Web-Connected CE Devices 274
Introduction 275
Digital Photography Ecosystem 277
AutoPhotobook System 279
Design-Driven Photo Selection and Pagination 283
Blurry Image Removal 283
Duplicate Photo Detection 284
Theme-Based Pagination and Layout 287
Artistic Background Resizing and Assignment 288
STArt Design for Automatic Resizable Artwork 289
Transformation Algorithm 290
Dynamic Photo Layout Region on the Page 292
Theme Grammar for Photobook 292
Automatic Layout 292
Prior Related Work 293
The AutoPhotobook Layout Engine 293
Results Illustrating Text Support 294
User Interface Design 295
Powering CE 2.0 with AutoPhotobook 298
Conclusion 301
References 301
Index 304