Multimodal Location Estimation of Videos and Images (eBook)

Dr. Gerald Friedland is the Director at the  Audio and Multimedia Research, International Computer Science InstituteDr. Jaeyoung Chois is a Researcher at the Audio and Multimedia Research, International Computer Science Institute

Preface 5
Acknowledgments 7
Contents 9
Contributors 11
1 Introduction 13
References 16
2 The Benchmark as a Research Catalyst: Charting the Progress of Geo-prediction for Social Multimedia 17
2.1 Introduction 18
2.1.1 The Placing Challenge for Social Multimedia 19
2.1.2 The Benefits of Benchmarking 21
2.2 Charting the Progress 23
2.2.1 Placing Task 2010: Inception 24
2.2.2 Placing Task 2011: Consolidation 27
2.2.3 Placing Task 2012: Expansion 29
2.2.4 Placing Task 2013: Volume 31
2.2.5 Placing Task 2014: Horizons 35
2.3 Future Challenges for Geoprediction of Social Multimedia 36
2.3.1 Further Development of Definitions of ``Place'' 36
2.3.2 Definitions of the Task of Placing Social Multimedia 43
2.4 Conclusion and Outlook 45
2.4.1 Where Placing has Been 45
2.4.2 Where Placing Is Going 47
References 49
3 Large-Scale Image Geolocalization 53
3.1 Introduction 53
3.1.1 Background 55
3.1.2 Chapter Outline 56
3.2 Building a Geo-tagged Image Dataset 56
3.2.1 Evaluation Test Set 57
3.3 Simple, Baseline Geolocalization Method 58
3.3.1 Is the Data Helping? 59
3.3.2 Grouping Geolocation Estimates 60
3.4 Improving Geolocalization with More Features and Lazy Learning 60
3.4.1 Geometry Specific Color and Texton Histograms 61
3.4.2 Bags of SIFT Features 62
3.4.3 Geolocalization with Additional Features 63
3.4.4 Lazy Learning for Large-Scale Scene Geolocalization 63
3.4.5 Geolocalization Results with New Features and Lazy Learning 66
3.5 Why Does it Work? Deeper Performance Analysis 66
3.5.1 Measuring Performance Without Geographic Bias. 66
3.5.2 Measuring Category Level Geolocation Performance. 70
3.5.3 Measuring Landmark Geolocation Performance 71
3.6 Discussion 72
References 73
4 Vision-Based Fine-Grained Location Estimation 75
4.1 Landmark and Location Recognition 76
4.2 Image-Based Location Recognition 76
4.3 Estimating the Camera Viewing Direction 77
4.4 City-Scale Location Recognition 78
4.4.1 Large-Scale Image Database Indexing 79
4.4.2 Informative Codebook Generation 80
4.4.3 Geo-Visual Clutering 81
4.5 Location Estimation by 2D--3D Alignment 83
4.5.1 3D Model Reconstruction 83
4.5.2 Image Localization by View Registration 85
4.6 Accurate Mobile Visual Localization and Its Applications 87
4.6.1 Aerial-Imagery Matching 90
4.6.2 Intensity-Based Matching Through Dynamic Time Warping 92
4.6.3 Conclusions 93
References 93
5 Image-Based Positioning of Mobile Devices in Indoor Environments 96
5.1 Introduction 96
5.2 Database Preparation 98
5.3 Image Retrieval and Pose Estimation 102
5.4 Confidence Estimation 102
5.5 Experimental Results 106
5.6 Conclusion 109
References 109
6 Application of Large-Scale Classification Techniques for Simple Location Estimation Experiments 111
6.1 Introduction 111
6.2 Approaching the City-Verification Task 112
6.2.1 MFCC Acoustic Feature Extraction 113
6.2.2 Gaussian Mixture Modeling 114
6.2.3 GMM-SVM Approach 114
6.2.4 Language Modeling 117
6.2.5 Performance Evaluation 117
6.3 Related Work 117
6.4 Dataset 118
6.5 Experiments and Results 119
6.6 Discussion and Analysis 121
6.7 Conclusion and Future Work 122
References 122
7 Collaborative Multimodal Location Estimation of Consumer Media 124
7.1 Introduction 124
7.2 Literature Review 126
7.3 Data Sparsity 127
7.4 Graphical Models for Geo-tagging 128
7.5 Experimental Results 132
7.6 Conclusions and Future Work 134
References 135
8 Georeferencing Flickr Resources Based on Multimodal Features 136
8.1 Introduction 136
8.2 Data Selection and Preprocessing 138
8.2.1 Clustering the Training Data 139
8.2.2 Term Selection 142
8.3 Textual Approach 143
8.3.1 Variations on the Classification Approach 144
8.3.2 Variations on the Retrieval Approach 149
8.4 Visual Approach 152
8.4.1 A Classification Approach to Using Visual Information 153
8.4.2 A Retrieval Approach to Using Visual Information 155
8.5 Centroid-Based Candidate Fusion 156
8.6 Conclusion 159
References 160
9 Human Versus Machine: Establishing a Human Baseline for Multimodal Location Estimation 162
9.1 Introduction 163
9.2 Related Works 163
9.3 Task and Dataset 164
9.4 Establishing a Human Baseline 165
9.4.1 Qualification 166
9.4.2 The Web Interface 167
9.4.3 Collection of Human Intelligence 168
9.5 Machine-Based Location Estimation 168
9.5.1 Audio-Based Location Estimation 169
9.5.2 Visual Location Estimation 170
9.5.3 Multimodal Location Estimation 170
9.6 Results 173
9.7 Discussion 174
9.7.1 Machine Versus Human Using only Audio 174
9.7.2 Machine Versus Human Using Audio and Video 176
9.7.3 Machine Versus Human Using All Modalities 177
9.8 Conclusion and Future Work 179
References 179
10 Personalized Travel Navigation and Photo-Shooting Navigation Using Large-Scale Geotags 181
10.1 Introduction 181
10.2 Related Works 182
10.2.1 Travel Navigation 182
10.2.2 Photo-Shooting Navigation 183
10.2.3 Contribution of this Work 184
10.3 Travel Navigation 184
10.3.1 Inter-city Travel Navigation 184
10.3.2 Intra-city Travel Navigation 186
10.4 Photo-Shooting Navigation 187
10.5 Experimental Results 188
10.5.1 Experimental Setup 188
10.5.2 Inter-city Travel Navigation 189
10.5.3 Intra-city Travel Navigation 190
10.5.4 Photo-Shooting Navigation 196
10.6 Conclusions 198
References 198