Deep Learning for the Earth Sciences

Gustau Camps-Valls is Professor of Electrical Engineering and Lead Researcher in the Image Processing Laboratory (IPL) at the Universitat de València. His interests include development of statistical learning, mainly kernel machines and neural networks, for Earth sciences, from remote sensing to geoscience data analysis. Models efficiency and accuracy but also interpretability, consistency and causal discovery are driving his agenda on AI for Earth and climate. Devis Tuia, PhD, is Associate Professor at the Ecole Polytechnique Fédérale de Lausanne (EPFL). He leads the Environmental Computational Science and Earth Observation laboratory, which focuses on the processing of Earth observation data with computational methods to advance Environmental science. Xiao Xiang Zhu is Professor of Data Science in Earth Observation and Director of the Munich AI Future Lab AI4EO at the Technical University of Munich and heads the Department EO Data Science at the German Aerospace Center. Her lab develops innovative machine learning methods and big data analytics solutions to extract large scale geo-information from big Earth observation data, aiming at tackling societal grand challenges, e.g. Urbanization, UN’s SDGs and Climate Change. Markus Reichstein is Director of the Biogeochemical Integration Department at the Max-Planck- Institute for Biogeochemistry and Professor for Global Geoecology at the University of Jena. His main research interests include the response and feedback of ecosystems (vegetation and soils) to climatic variability with an Earth system perspective, considering coupled carbon, water and nutrient cycles. He has been tackling these topics with applied statistical learning for more than 15 years.

Foreword xvi
by Vipin Kumar, Regents Professor, University of Minnesota

Acknowledgments xvii

List of Contributors xviii

List of Acronyms xxiv

1 Introduction 1
Gustau Camps-Valls, Xiao Xiang Zhu, Devis Tuia, and Markus Reichstein

1.1 A Taxonomy of Deep Learning Approaches 2

1.2 Deep Learning in Remote Sensing 3

1.3 Deep Learning in Geosciences and Climate 7

1.4 Book Structure and Roadmap 9

Part I Deep Learning to Extract Information from Remote Sensing Images 13

2 Learning Unsupervised Feature Representations of Remote Sensing Data with Sparse Convolutional Networks 15
Jose E. Adsuara, Manuel Campos-Taberner, Javier García-Haro, Carlo Gatta, Adriana Romero, and Gustau Camps-Valls

2.1 Introduction 15

2.2 Sparse Unsupervised Convolutional Networks 17

2.2.1 Sparsity as the Guiding Criterion 17

2.2.2 The EPLS Algorithm 18

2.2.3 Remarks 18

2.3 Applications 19

2.3.1 Hyperspectral Image Classification 19

2.3.2 Multisensor Image Fusion 21

2.4 Conclusions 22

3 Generative Adversarial Networks in the Geosciences 24
Gonzalo Mateo-García, Valero Laparra, Christian Requena-Mesa, and Luis Gómez-Chova

3.1 Introduction 24

3.2 Generative Adversarial Networks 25

3.2.1 Unsupervised GANs 25

3.2.2 Conditional GANs 26

3.2.3 Cycle-consistent GANs 27

3.3 GANs in Remote Sensing and Geosciences 28

3.3.1 GANs in Earth Observation 28

3.3.2 Conditional GANs in Earth Observation 30

3.3.3 CycleGANs in Earth Observation 30

3.4 Applications of GANs in Earth Observation 31

3.4.1 Domain Adaptation Across Satellites 31

3.4.2 Learning to Emulate Earth Systems from Observations 33

3.5 Conclusions and Perspectives 36

4 Deep Self-taught Learning in Remote Sensing 37
Ribana Roscher

4.1 Introduction 37

4.2 Sparse Representation 38

4.2.1 Dictionary Learning 39

4.2.2 Self-taught Learning 40

4.3 Deep Self-taught Learning 40

4.3.1 Application Example 43

4.3.2 Relation to Deep Neural Networks 44

4.4 Conclusion 45

5 Deep Learning-based Semantic Segmentation in Remote Sensing 46
Devis Tuia, Diego Marcos, Konrad Schindler, and Bertrand Le Saux

5.1 Introduction 46

5.2 Literature Review 47

5.3 Basics on Deep Semantic Segmentation: Computer Vision Models 49

5.3.1 Architectures for Image Data 49

5.3.2 Architectures for Point-clouds 52

5.4 Selected Examples 55

5.4.1 Encoding Invariances to Train Smaller Models: The example of Rotation 55

5.4.2 Processing 3D Point Clouds as a Bundle of Images: SnapNet 59

5.4.3 Lake Ice Detection from Earth and from Space 62

5.5 Concluding Remarks 66

6 Object Detection in Remote Sensing 67
Jian Ding, Jinwang Wang, Wen Yang, and Gui-Song Xia

6.1 Introduction 67

6.1.1 Problem Description 67

6.1.2 Problem Settings of Object Detection 69

6.1.3 Object Representation in Remote Sensing 69

6.1.4 Evaluation Metrics 69

6.1.4.1 Precision-Recall Curve 70

6.1.4.2 Average Precision and Mean Average Precision 71

6.1.5 Applications 71

6.2 Preliminaries on Object Detection with Deep Models 72

6.2.1 Two-stage Algorithms 72

6.2.1.1 R-CNNs 72

6.2.1.2 R-fcn 73

6.2.2 One-stage Algorithms 73

6.2.2.1 Yolo 73

6.2.2.2 Ssd 73

6.3 Object Detection in Optical RS Images 75

6.3.1 Related Works 75

6.3.1.1 Scale Variance 75

6.3.1.2 Orientation Variance 75

6.3.1.3 Oriented Object Detection 75

6.3.1.4 Detecting in Large-size Images 76

6.3.2 Datasets and Benchmark 77

6.3.2.1 Dota 77

6.3.2.2 VisDrone 77

6.3.2.3 Dior 77

6.3.2.4 xView 77

6.3.3 Two Representative Object Detectors in Optical RS Images 78

6.3.3.1 Mask OBB 78

6.3.3.2 RoI Transformer 82

6.4 Object Detection in SAR Images 86

6.4.1 Challenges of Detection in SAR Images 86

6.4.2 Related Works 86

6.4.3 Datasets and Benchmarks 88

6.5 Conclusion 89

7 Deep Domain Adaptation in Earth Observation 90
Benjamin Kellenberger, Onur Tasar, Bharath Bhushan Damodaran, Nicolas Courty, and Devis Tuia

7.1 Introduction 90

7.2 Families of Methodologies 91

7.3 Selected Examples 93

7.3.1 Adapting the Inner Representation 93

7.3.2 Adapting the Inputs Distribution 97

7.3.3 Using (few, well chosen) Labels from the Target Domain 100

7.4 Concluding remarks 104

8 Recurrent Neural Networks and the Temporal Component 105
Marco Körner and Marc Rußwurm

8.1 Recurrent Neural Networks 106

8.1.1 Training RNNs 107

8.1.1.1 Exploding and Vanishing Gradients 107

8.1.1.2 Circumventing Exploding and Vanishing Gradients 109

8.2 Gated Variants of RNNs 111

8.2.1 Long Short-term Memory Networks 111

8.2.1.1 The Cell State c t and the Hidden State h t 112

8.2.1.2 The Forget Gate f t 112

8.2.1.3 The Modulation Gate V T and the Input Gate I T 112

8.2.1.4 The Output Gate o t 112

8.2.1.5 Training LSTM Networks 113

8.2.2 Other Gated Variants 113

8.3 Representative Capabilities of Recurrent Networks 114

8.3.1 Recurrent Neural Network Topologies 114

8.3.2 Experiments 115

8.4 Application in Earth Sciences 117

8.5 Conclusion 118

9 Deep Learning for Image Matching and Co-registration 120
Maria Vakalopoulou, Stergios Christodoulidis, Mihir Sahasrabudhe, and Nikos Paragios

9.1 Introduction 120

9.2 Literature Review 123

9.2.1 Classical Approaches 123

9.2.2 Deep Learning Techniques for Image Matching 124

9.2.3 Deep Learning Techniques for Image Registration 125

9.3 Image Registration with Deep Learning 126

9.3.1 2D Linear and Deformable Transformer 126

9.3.2 Network Architectures 127

9.3.3 Optimization Strategy 128

9.3.4 Dataset and Implementation Details 129

9.3.5 Experimental Results 129

9.4 Conclusion and Future Research 134

9.4.1 Challenges and Opportunities 134

9.4.1.1 Dataset with Annotations 134

9.4.1.2 Dimensionality of Data 135

9.4.1.3 Multitemporal Datasets 135

9.4.1.4 Robustness to Changed Areas 135

10 Multisource Remote Sensing Image Fusion 136
Wei He, Danfeng Hong, Giuseppe Scarpa, Tatsumi Uezato, and Naoto Yokoya

10.1 Introduction 136

10.2 Pansharpening 137

10.2.1 Survey of Pansharpening Methods Employing Deep Learning 137

10.2.2 Experimental Results 140

10.2.2.1 Experimental Design 140

10.2.2.2 Visual and Quantitative Comparison in Pansharpening 140

10.3 Multiband Image Fusion 143

10.3.1 Supervised Deep Learning-based Approaches 143

10.3.2 Unsupervised Deep Learning-based Approaches 145

10.3.3 Experimental Results 146

10.3.3.1 Comparison Methods and Evaluation Measures 146

10.3.3.2 Dataset and Experimental Setting 146

10.3.3.3 Quantitative Comparison and Visual Results 147

10.4 Conclusion and Outlook 148

11 Deep Learning for Image Search and Retrieval in Large Remote Sensing Archives 150
Gencer Sumbul, Jian Kang, and Begüm Demir

11.1 Introduction 150

11.2 Deep Learning for RS CBIR 152

11.3 Scalable RS CBIR Based on Deep Hashing 156

11.4 Discussion and Conclusion 159

Acknowledgement 160

Part II Making a Difference in the Geosciences with Deep Learning 161

12 Deep Learning for Detecting Extreme Weather Patterns 163
Mayur Mudigonda, Prabhat Ram, Karthik Kashinath, Evan Racah, Ankur Mahesh, Yunjie Liu, Christopher Beckham, Jim Biard, Thorsten Kurth, Sookyung Kim, Samira Kahou, Tegan Maharaj, Burlen Loring, Christopher Pal, Travis O’Brien, Kenneth E. Kunkel, Michael F. Wehner, and William D. Collins

12.1 Scientific Motivation 163

12.2 Tropical Cyclone and Atmospheric River Classification 166

12.2.1 Methods 166

12.2.2 Network Architecture 167

12.2.3 Results 169

12.3 Detection of Fronts 170

12.3.1 Analytical Approach 170

12.3.2 Dataset 171

12.3.3 Results 172

12.3.4 Limitations 174

12.4 Semi-supervised Classification and Localization of Extreme Events 175

12.4.1 Applications of Semi-supervised Learning in Climate Modeling 175

12.4.1.1 Supervised Architecture 176

12.4.1.2 Semi-supervised Architecture 176

12.4.2 Results 176

12.4.2.1 Frame-wise Reconstruction 176

12.4.2.2 Results and Discussion 178

12.5 Detecting Atmospheric Rivers and Tropical Cyclones Through Segmentation Methods 179

12.5.1 Modeling Approach 179

12.5.1.1 Segmentation Architecture 180

12.5.1.2 Climate Dataset and Labels 181

12.5.2 Architecture Innovations: Weighted Loss and Modified Network 181

12.5.3 Results 183

12.6 Challenges and Implications for the Future 184

12.7 Conclusions 185

13 Spatio-temporal Autoencoders in Weather and Climate Research 186
Xavier-Andoni Tibau, Christian Reimers, Christian Requena-Mesa, and Jakob Runge

13.1 Introduction 186

13.2 Autoencoders 187

13.2.1 A Brief History of Autoencoders 188

13.2.2 Archetypes of Autoencoders 189

13.2.3 Variational Autoencoders (VAE) 191

13.2.4 Comparison Between Autoencoders and Classical Methods 192

13.3 Applications 193

13.3.1 Use of the Latent Space 193

13.3.1.1 Reduction of Dimensionality for the Understanding of the System Dynamics and its Interactions 195

13.3.1.2 Dimensionality Reduction for Feature Extraction and Prediction 199

13.3.2 Use of the Decoder 199

13.3.2.1 As a Random Sample Generator 201

13.3.2.2 Anomaly Detection 201

13.3.2.3 Use of a Denoising Autoencoder (DAE) Decoder 202

13.4 Conclusions and Outlook 203

14 Deep Learning to Improve Weather Predictions 204
Peter D. Dueben, Peter Bauer, and Samantha Adams

14.1 Numerical Weather Prediction 204

14.2 How Will Machine Learning Enhance Weather Predictions? 207

14.3 Machine Learning Across the Workflow of Weather Prediction 208

14.4 Challenges for the Application of ML in Weather Forecasts 213

14.5 The Way Forward 216

15 Deep Learning and the Weather Forecasting Problem: Precipitation Nowcasting 218
Zhihan Gao, Xingjian Shi, Hao Wang, Dit-Yan Yeung, Wang-chun Woo, and Wai-Kin Wong

15.1 Introduction 218

15.2 Formulation 220

15.3 Learning Strategies 221

15.4 Models 223

15.4.1 FNN-based Odels 223

15.4.2 RNN-based Models 225

15.4.3 Encoder-forecaster Structure 226

15.4.4 Convolutional LSTM 226

15.4.5 ConvLSTM with Star-shaped Bridge 227

15.4.6 Predictive RNN 228

15.4.7 Memory in Memory Network 229

15.4.8 Trajectory GRU 231

15.5 Benchmark 233

15.5.1 HKO-7 Dataset 234

15.5.2 Evaluation Methodology 234

15.5.3 Evaluated Algorithms 235

15.5.4 Evaluation Results 236

15.6 Discussion 236

Appendix 238

Acknowledgement 239

16 Deep Learning for High-dimensional Parameter Retrieval 240
David Malmgren-Hansen

16.1 Introduction 240

16.2 Deep Learning Parameter Retrieval Literature 242

16.2.1 Land 242

16.2.2 Ocean 243

16.2.3 Cryosphere 244

16.2.4 Global Weather Models 244

16.3 The Challenge of High-dimensional Problems 244

16.3.1 Computational Load of CNNs 247

16.3.2 Mean Square Error or Cross-entropy Optimization? 249

16.4 Applications and Examples 250

16.4.1 Utilizing High-Dimensional Spatio-spectral Information with CNNs 250

16.4.2 The Effect of Loss Functions in Retrieval of Sea Ice Concentrations 253

16.5 Conclusion 257

17 A Review of Deep Learning for Cryospheric Studies 258
Lin Liu

17.1 Introduction 258

17.2 Deep-learning-based Remote Sensing Studies of the Cryosphere 260

17.2.1 Glaciers 260

17.2.2 Ice Sheet 261

17.2.3 Snow 262

17.2.4 Permafrost 263

17.2.5 Sea Ice 264

17.2.6 River Ice 265

17.3 Deep-learning-based Modeling of the Cryosphere 265

17.4 Summary and Prospect 266

Appendix: List of Data and Codes 267

18 Emulating Ecological Memory with Recurrent Neural Networks 269
Basil Kraft, Simon Besnard, and Sujan Koirala

18.1 Ecological Memory Effects: Concepts and Relevance 269

18.2 Data-driven Approaches for Ecological memory Effects 270

18.2.1 A Brief Overview of Memory Effects 270

18.2.2 Data-driven Methods for Memory Effects 271

18.3 Case Study: Emulating a Physical Model Using Recurrent Neural Networks 272

18.3.1 Physical Model Simulation Data 272

18.3.2 Experimental Design 273

18.3.3 RNN Setup and Training 274

18.4 Results and Discussion 276

18.4.1 The Predictive Capability Across Scales 276

18.4.2 Prediction of Seasonal Dynamics 279

18.5 Conclusions 281

Part III Linking Physics and Deep Learning Models 283

19 Applications of Deep Learning in Hydrology 285
Chaopeng Shen and Kathryn Lawson

19.1 Introduction 285

19.2 Deep Learning Applications in Hydrology 286

19.2.1 Dynamical System Modeling 286

19.2.1.1 Large-scale Hydrologic Modeling with Big Data 286

19.2.1.2 Data-limited LSTM Applications 290

19.2.2 Physics-constrained Hydrologic Machine Learning 292

19.2.3 Information Retrieval for Hydrology 293

19.2.4 Physically-informed Machine Learning for Subsurface Flow and Reactive Transport Modeling 294

19.2.5 Additional Observations 296

19.3 Current Limitations and Outlook 296

20 Deep Learning of Unresolved Turbulent Ocean Processes in Climate Models 298
Laure Zanna and Thomas Bolton

20.1 Introduction 298

20.2 The Parameterization Problem 299

20.3 Deep Learning Parameterizations of Subgrid Ocean Processes 300

20.3.1 Why DL for Subgrid Parameterizations? 300

20.3.2 Recent Advances in DL for Subgrid Parameterizations 300

20.4 Physics-aware Deep Learning 301

20.5 Further Challenges ahead for Deep Learning Parameterizations 303

21 Deep Learning for the Parametrization of Subgrid Processes in Climate Models 307
Pierre Gentine, Veronika Eyring, and Tom Beucler

21.1 Introduction 307

21.2 Deep Neural Networks for Moist Convection (Deep Clouds) Parametrization 309

21.3 Physical Constraints and Generalization 312

21.4 Future Challenges 314

22 Using Deep Learning to Correct Theoretically-derived Models 315
PeterA.G.Watson

22.1 Experiments with the Lorenz ’96 System 317

22.1.1 The Lorenz’96 Equations and Coarse-scale Models 318

22.1.1.1 Theoretically-derived Coarse-scale Model 318

22.1.1.2 Models with ANNs 319

22.1.2 Results 320

22.1.2.1 Single-timestep Tendency Prediction Errors 320

22.1.2.2 Forecast and Climate Prediction Skill 321

22.1.3 Testing Seamless Prediction 324

22.2 Discussion and Outlook 324

22.2.1 Towards Earth System Modeling 325

22.2.2 Application to Climate Change Studies 326

22.3 Conclusion 327

23 Outlook 328
Markus Reichstein, Gustau Camps-Valls, Devis Tuia, and Xiao Xiang Zhu

Bibliography 331

Index 401