Deep Learning for the Earth Sciences: A Comprehensive Approach to Remote Sensing, Climate Science and Geosciences

Foreword xvii Acknowledgments xix List of Contributors xxi List of Acronyms xxvii 1 Introduction 1 Gustau Camps-Valls, Xiao Xiang Zhu, Devis Tuia, and Markus Reichstein 1.1 A Taxonomy of Deep Learning Approaches 2 1.2 Deep Learning in Remote Sensing 3 1.3 Deep Learning in Geosciences and Climate 7 1.4 Book Structure and Roadmap 9 Part I Deep Learning to Extract Information from Remote Sensing Images 13 2 Learning Unsupervised Feature Representations of Remote Sensing Data with Sparse Convolutional Networks 15 Jose E. Adsuara, Manuel Campos-Taberner, Javier García-Haro, Carlo Gatta, Adriana Romero, and Gustau Camps-Valls 2.1 Introduction 15 2.2 Sparse Unsupervised Convolutional Networks 17 2.2.1 Sparsity as the Guiding Criterion 17 2.2.2 The EPLS Algorithm 18 2.2.3 Remarks 18 2.3 Applications 19 2.3.1 Hyperspectral Image Classification 19 2.3.2 Multisensor Image Fusion 21 2.4 Conclusions 22 3 Generative Adversarial Networks in the Geosciences 24 Gonzalo Mateo-García, Valero Laparra, Christian Requena-Mesa, and Luis Gómez-Chova 3.1 Introduction 24 3.2 Generative Adversarial Networks 25 3.2.1 Unsupervised GANs 25 3.2.2 Conditional GANs 26 3.2.3 Cycle-consistent GANs 27 3.3 GANs in Remote Sensing and Geosciences 28 3.3.1 GANs in Earth Observation 28 3.3.2 Conditional GANs in Earth Observation 30 3.3.3 CycleGANs in Earth Observation 30 3.4 Applications of GANs in Earth Observation 31 3.4.1 Domain Adaptation Across Satellites 31 3.4.2 Learning to Emulate Earth Systems from Observations 33 3.5 Conclusions and Perspectives 36 4 Deep Self-taught Learning in Remote Sensing 37 Ribana Roscher 4.1 Introduction 37 4.2 Sparse Representation 38 4.2.1 Dictionary Learning 39 4.2.2 Self-taught Learning 40 4.3 Deep Self-taught Learning 40 4.3.1 Application Example 43 4.3.2 Relation to Deep Neural Networks 44 4.4 Conclusion 45 5 Deep Learning-based Semantic Segmentation in Remote Sensing 46 Devis Tuia, Diego Marcos, Konrad Schindler, and Bertrand Le Saux 5.1 Introduction 46 5.2 Literature Review 47 5.3 Basics on Deep Semantic Segmentation: Computer Vision Models 49 5.3.1 Architectures for Image Data 49 5.3.2 Architectures for Point-clouds 52 5.4 Selected Examples 55 5.4.1 Encoding Invariances to Train Smaller Models: The example of Rotation 55 5.4.2 Processing 3D Point Clouds as a Bundle of Images: SnapNet 59 5.4.3 Lake Ice Detection from Earth and from Space 62 5.5 Concluding Remarks 66 6 Object Detection in Remote Sensing 67 Jian Ding, Jinwang Wang, Wen Yang, and Gui-Song Xia 6.1 Introduction 67 6.1.1 Problem Description 67 6.1.2 Problem Settings of Object Detection 69 6.1.3 Object Representation in Remote Sensing 69 6.1.4 Evaluation Metrics 69 6.1.4.1 Precision-recall Curve 70 6.1.4.2 Average Precision and Mean Average Precision 71 6.1.5 Applications 71 6.2 Preliminaries on Object Detection with Deep Models 72 6.2.1 Two-stage Algorithms 72 6.2.1.1 R-CNNs 72 6.2.1.2 R-FCN 73 6.2.2 One-stage Algorithms 73 6.2.2.1 YOLO 73 6.2.2.2 SSD 73 6.3 Object Detection in Optical RS Images 75 6.3.1 RelatedWorks 75 6.3.1.1 Scale Variance 75 6.3.1.2 Orientation Variance 75 6.3.1.3 Oriented Object Detection 75 6.3.1.4 Detecting in Large-size Images 76 6.3.2 Datasets and Benchmark 77 6.3.2.1 DOTA 77 6.3.2.2 VisDrone 77 6.3.2.3 DIOR 77 6.3.2.4 xView 77 6.3.3 Two Representative Object Detectors in Optical RS Images 78 6.3.3.1 Mask OBB 78 6.3.3.2 RoI Transformer 82 6.4 Object Detection in SAR Images 86 6.4.1 Challenges of Detection in SAR Images 86 6.4.2 RelatedWorks 86 6.4.3 Datasets and Benchmarks 88 6.5 Conclusion 89 7 Deep Domain adaptation in Earth Observation 90 Benjamin Kellenberger, Onur Tasar, Bharath Bhushan Damodaran, Nicolas Courty, and Devis Tuia 7.1 Introduction 90 7.2 Families of Methodologies 91 7.3 Selected Examples 93 7.3.1 Adapting the Inner Representation 93 7.3.2 Adapting the Inputs Distribution 97 7.3.3 Using (few, well chosen) Labels from the Target Domain 100 7.4 Concluding remarks 104 8 Recurrent Neural Networks and the Temporal Component 105 Marco Körner and Marc Rußwurm 8.1 Recurrent Neural Networks 106 8.1.1 Training RNNs 107 8.1.1.1 Exploding and Vanishing Gradients 107 8.1.1.2 Circumventing Exploding and Vanishing Gradients 109 8.2 Gated Variants of RNNs 111 8.2.1 Long Short-term Memory Networks 111 8.2.1.1 The Cell State ct and the Hidden State ht 112 8.2.1.2 The Forget Gate ft 112 8.2.1.3 The Modulation Gate vt and the Input Gate it 112 8.2.1.4 The Output Gate ot 112 8.2.1.5 Training LSTM Networks 113 8.2.2 Other Gated Variants 113 8.3 Representative Capabilities of Recurrent Networks 114 8.3.1 Recurrent Neural Network Topologies 114 8.3.2 Experiments 115 8.4 Application in Earth Sciences 117 8.5 Conclusion 118 9 Deep Learning for Image Matching and Co-registration 120 Maria Vakalopoulou, Stergios Christodoulidis, Mihir Sahasrabudhe, and Nikos Paragios 9.1 Introduction 120 9.2 Literature Review 123 9.2.1 Classical Approaches 123 9.2.2 Deep Learning Techniques for Image Matching 124 9.2.3 Deep Learning Techniques for Image Registration 125 9.3 Image Registration with Deep Learning 126 9.3.1 2D Linear and Deformable Transformer 126 9.3.2 Network Architectures 127 9.3.3 Optimization Strategy 128 9.3.4 Dataset and Implementation Details 129 9.3.5 Experimental Results 129 9.4 Conclusion and Future Research 134 9.4.1 Challenges and Opportunities 134 9.4.1.1 Dataset with Annotations 134 9.4.1.2 Dimensionality of Data 135 9.4.1.3 Multitemporal Datasets 135 9.4.1.4 Robustness to Changed Areas 135 10 Multisource Remote Sensing Image Fusion 136 Wei He, Danfeng Hong, Giuseppe Scarpa, Tatsumi Uezato, and Naoto Yokoya 10.1 Introduction 136 10.2 Pansharpening 137 10.2.1 Survey of Pansharpening Methods Employing Deep Learning 137 10.2.2 Experimental Results 140 10.2.2.1 Experimental Design 140 10.2.2.2 Visual and Quantitative Comparison in Pansharpening 140 10.3 Multiband Image Fusion 143 10.3.1 Supervised Deep Learning-based Approaches 143 10.3.2 Unsupervised Deep Learning-based Approaches 145 10.3.3 Experimental Results 146 10.3.3.1 Comparison Methods and Evaluation Measures 146 10.3.3.2 Dataset and Experimental Setting 146 10.3.3.3 Quantitative Comparison and Visual Results 147 10.4 Conclusion and Outlook 148 11 Deep Learning for Image Search and Retrieval in Large Remote Sensing Archives 150 Gencer Sumbul, Jian Kang, and Begüm Demir 11.1 Introduction 150 11.2 Deep Learning for RS CBIR 152 11.3 Scalable RS CBIR Based on Deep Hashing 156 11.4 Discussion and Conclusion 160 Part II Making a Difference in the Geosciences With Deep Learning 161 12 Deep Learning for Detecting Extreme Weather Patterns 163 Mayur Mudigonda, Prabhat, Karthik Kashinath, Evan Racah, Ankur Mahesh, Yunjie Liu, Christopher Beckham, Jim Biard, Thorsten Kurth, Sookyung Kim, Samira Kahou, Tegan Maharaj, Burlen Loring, Christopher Pal, Travis O'Brien, Ken Kunkel, Michael F. Wehner, and William D. Collins 12.1 Scientific Motivation 163 12.2 Tropical Cyclone and Atmospheric River Classification 166 12.2.1 Methods 166 12.2.2 Network Architecture 167 12.2.3 Results 169 12.3 Detection of Fronts 170 12.3.1 Analytical Approach 170 12.3.2 Dataset 171 12.3.3 Results 172 12.3.4 Limitations 174 12.4 Semi-supervised Classification and Localization of Extreme Events 175 12.4.1 Applications of Semi-supervised Learning in Climate Modeling 175 12.4.1.1 Supervised Architecture 176 12.4.1.2 Semi-supervised Architecture 176 12.4.2 Results 176 12.4.2.1 Frame-wise Reconstruction 176 12.4.2.2 Results and Discussion 178 12.5 Detecting Atmospheric Rivers and Tropical Cyclones Through Segmentation Methods 179 12.5.1 Modeling Approach 179 12.5.1.1 Segmentation Architecture 180 12.5.1.2 Climate Dataset and Labels 181 12.5.2 Architecture Innovations:Weighted Loss and Modified Network 181 12.5.3 Results 183 12.6 Challenges and Implications for the Future 184 12.7 Conclusions 185 13 Spatio-temporal Autoencoders in Weather and Climate Research 186 Xavier-Andoni Tibau, Christian Reimers, Christian Requena-Mesa, and Jakob Runge 13.1 Introduction 186 13.2 Autoencoders 187 13.2.1 A Brief History of Autoencoders 188 13.2.2 Archetypes of Autoencoders 189 13.2.3 Variational Autoencoders (VAE) 191 13.2.4 Comparison Between Autoencoders and Classical Methods 192 13.3 Applications 193 13.3.1 Use of the Latent Space 193 13.3.1.1 Reduction of Dimensionality for the Understanding of the System Dynamics and its Interactions 195 13.3.1.2 Dimensionality Reduction for Feature Extraction and Prediction 199 13.3.2 Use of the Decoder 199 13.3.2.1 As a Random Sample Generator 201 13.3.2.2 Anomaly Detection 201 13.3.2.3 Use of a Denoising Autoencoder (DAE) Decoder 202 13.4 Conclusions and Outlook 203 14 Deep Learning to Improve Weather Predictions 204 Peter D. Dueben, Peter Bauer, and Samantha Adams 14.1 NumericalWeather Prediction 204 14.2 How Will Machine Learning EnhanceWeather Predictions? 207 14.3 Machine Learning Across theWorkflow ofWeather Prediction 208 14.4 Challenges for the Application of ML inWeather Forecasts 213 14.5 TheWay Forward 216 15 Deep Learning and the Weather Forecasting Problem: Precipitation Nowcasting 218 Zhihan Gao, Xingjian Shi, Hao Wang, Dit-Yan Yeung, Wang-chun Woo, and Wai-Kin Wong 15.1 Introduction 218 15.2 Formulation 220 15.3 Learning Strategies 221 15.4 Models 223 15.4.1 FNN-based Odels 223 15.4.2 RNN-based Models 225 15.4.3 Encoder-forecaster Structure 226 15.4.4 Convolutional LSTM 226 15.4.5 ConvLSTM with Star-shaped Bridge 227 15.4.6 Predictive RNN 228 15.4.7 Memory in Memory Network 229 15.4.8 Trajectory GRU 231 15.5 Benchmark 233 15.5.1 HKO-7 Dataset 234 15.5.2 Evaluation Methodology 234 15.5.3 Evaluated Algorithms 235 15.5.4 Evaluation Results 236 15.6 Discussion 236 Appendix 238 Acknowledgement 239 16 Deep Learning for High-dimensional Parameter Retrieval 240 David Malmgren-Hansen 16.1 Introduction 240 16.2 Deep Learning Parameter Retrieval Literature 242 16.2.1 Land 242 16.2.2 Ocean 243 16.2.3 Cryosphere 244 16.2.4 GlobalWeather Models 244 16.3 The Challenge of High-dimensional Problems 244 16.3.1 Computational Load of CNNs 247 16.3.2 Mean Square Error or Cross-Entropy Optimization? 249 16.4 Applications and Examples 250 16.4.1 Utilizing High-Dimensional Spatio-spectral Information with CNNs 250 16.4.2 The Effect of Loss Functions in Retrieval of Sea Ice Concentrations 253 16.5 Conclusion 257 17 A Review of Deep Learning for Cryospheric Studies 258 Lin Liu 17.1 Introduction 258 17.2 Deep-learning-based Remote Sensing Studies of the Cryosphere 260 17.2.1 Glaciers 260 17.2.2 Ice Sheet 261 17.2.3 Snow 262 17.2.4 Permafrost 263 17.2.5 Sea Ice 264 17.2.6 River Ice 265 17.3 Deep-learning-based Modeling of the Cryosphere 265 17.4 Summary and Prospect 266 Appendix: List of data and codes 267 18 Emulating Ecological Memory with Recurrent Neural Networks 269 Basil Kraft, Simon Besnard, and Sujan Koirala 18.1 Ecological Memory Effects: Concepts and Relevance 269 18.2 Data-driven Approaches for Ecological memory Effects 270 18.2.1 A Brief Overview of Memory Effects 270 18.2.2 Data-driven Methods for Memory Effects 271 18.3 Case Study: Emulating a Physical Model Using Recurrent Neural Networks 272 18.3.1 Physical Model Simulation Data 272 18.3.2 Experimental Design 273 18.3.3 RNN Setup and Training 274 18.4 Results and Discussion 276 18.4.1 The predictive capability across scales 276 18.4.2 Prediction of Seasonal Dynamics 279 18.5 Conclusions 281 Part III Linking Physics and Deep Learning Models 283 19 Applications of Deep Learning in Hydrology 285 Chaopeng Shen and Kathryn Lawson 19.1 Introduction 285 19.2 Deep Learning Applications in Hydrology 286 19.2.1 Dynamical System Modeling 286 19.2.1.1 Large-scale Hydrologic Modeling with Big Data 286 19.2.1.2 Data-limited LSTM Applications 289 19.2.2 Physics-constrained Hydrologic Machine Learning 292 19.2.3 Information Retrieval for Hydrology 293 19.2.4 Physically-informed Machine Learning for Subsurface Flow and Reactive Transport Modeling 294 19.2.5 Additional Observations 296 19.3 Current Limitations and Outlook 296 20 Deep Learning of Unresolved Turbulent Ocean Processes in Climate Models 298 Laure Zanna and Thomas Bolton 20.1 Introduction 298 20.2 The Parameterization Problem 299 20.3 Deep Learning Parameterizations of Subgrid Ocean Processes 300 20.3.1 Why DL for Subgrid Parameterizations? 300 20.3.2 Recent Advances in DL for Subgrid Parameterizations 300 20.4 Physics-aware Deep Learning 301 20.5 Further Challenges ahead for Deep Learning Parameterizations 303 21 Deep Learning for the Parametrization of Subgrid Processes in Climate Models 307 Pierre Gentine, Veronika Eyring, and Tom Beucler 21.1 Introduction 307 21.2 Deep Neural Networks for Moist Convection (Deep Clouds) Parametrization 309 21.3 Physical Constraints and Generalization 312 21.4 Future Challenges 314 22 Using Deep Learning to Correct Theoretically-Derived Models 315 Peter A. G. Watson 22.1 Experiments with the Lorenz '96 System 317 22.1.1 The Lorenz'96 Equations and Coarse-Scale Models 318 22.1.1.1 Theoretically-derived Coarse-Scale Model 318 22.1.1.2 Models with ANNs 319 22.1.2 Results 320 22.1.2.1 Single-timestep Tendency Prediction Errors 320 22.1.2.2 Forecast and Climate Prediction Skill 321 22.1.3 Testing Seamless Prediction 324 22.2 Discussion and Outlook 324 22.2.1 Towards Earth System Modeling 325 22.2.2 Application to Climate Change Studies 326 22.3 Conclusion 327 23 Outlook 328 Markus Reichstein, Gustau Camps-Valls, Devis Tuia, and Xiao Xiang Zhu Bibliography 331 Index 409