Methods and Techniques in Deep Learning
Description
Introduces multiple state-of-the-art deep learning architectures for mmWave radar in a variety of advanced applications
Methods and Techniques in Deep Learning: Advancements in mmWave Radar Solutions provides a timely and authoritative overview of the use of artificial intelligence (AI)-based processing for various mmWave radar applications. Focusing on practical deep learning techniques, this comprehensive volume explains the fundamentals of deep learning, reviews cutting-edge deep metric learning techniques, describes different typologies of reinforcement learning (RL) algorithms, highlights how domain adaptation (DA) can be used for improving the performance of machine learning (ML) algorithms, and more. Throughout the book, readers are exposed to product-ready deep learning solutions while learning skills that are relevant for building any industrial-grade, sensor-based deep learning solution.
A team of authors with more than 70 filed patents and 100 published papers on AI and sensor processing illustrates how deep learning is enabling a range of advanced industrial, consumer, and automotive applications of mmWave radars. In-depth chapters cover topics including multi-modal deep learning approaches, the elemental blocks required to formulate Bayesian deep learning, how domain adaptation (DA) can be used for improving the performance of machine learning algorithms, and geometric deep learning are used for processing point clouds. In addition, the book:
* Discusses various advanced applications and how their respective challenges have been addressed using different deep learning architectures and algorithms
* Describes deep learning in the context of computer vision, natural language processing, sensor processing, and mmWave radar sensors
* Demonstrates how deep parametric learning reduces the number of trainable parameters and improves the data flow
* Presents several human-machine interface (HMI) applications such as gesture recognition, human activity classification, human localization and tracking, in-cabin automotive occupancy sensing
Methods and Techniques in Deep Learning: Advancements in mmWave Radar Solutions is an invaluable resource for industry professionals, researchers, and graduate students working in systems engineering, signal processing, sensors, data science, and AI.
More details
Other editions
Additional editions
Persons
Avik Santra is Head of Advanced Artificial Intelligence at Infineon Technologies, Munich, Germany.
Souvik Hazra is a Senior Staff Machine Learning Engineer at Infineon Technologies, Munich, Germany.
Lorenzo Servadei is a Senior Staff Machine Learning Engineer at Infineon Technologies and a Lecturer at The Technical University of Munich (TU München), Germany.
Thomas Stadelmayer is a Staff Machine Learning Engineer at Infineon Technologies, Munich, Germany.
Michael Stephan is a PhD candidate at Infineon Technologies, Munich, Germany and Friedrich-Alexander-University of Erlangen-Nürnberg, Germany.
Anand Dubey is a Staff Machine Learning Engineer at Infineon Technologies.
Content
Preface
Acronyms
1 Introduction to Radar Processing & Deep Learning 1
1.1 Basics of Radar Systems 1
1.1.1 Fundamentals 2
1.1.2 Signal Modulation 2
1.2 FMCW Signal Processing 6
1.2.1 Frequency-Domain Analysis 7
1.3 Target Detection & Clustering 14
1.4 Target Tracking 19
1.4.1 Track Management 21
1.4.2 Track Filtering 22
1.5 Target Representation 28
1.5.1 Image Representation 30
1.5.2 Point-Cloud Maps 34
1.6 Target Recognition 36
1.6.1 Feedforward Network 37
1.6.2 Convolutional Neural Networks (CNN) 37
1.6.3 Recurrent Neural Network (RNN) 43
1.6.4 Autoencoder & Variational Autoencoder 47
1.6.5 Generative Adversial Network 51
1.6.6 Transformer 54
1.7 Training a Neural Network 56
1.7.1 Forward Pass & Backpropagation 57
1.7.2 Optimizers 62
1.7.3 Loss Functions 65
1.8 Questions to the Reader 66
Bibliography 68
2 Deep Metric Learning 75
2.1 Introduction 78
2.2 Pairwise methods 79
2.2.1 Contrastive Loss 79
2.2.2 Triplet Loss 80
2.2.3 Quadruplet Loss 81
2.2.4 N-Pair Loss 82
2.2.5 Big Picture 83
2.3 End-to-end Learning 84
2.3.1 Cosine Similarity 86
2.3.2 Euclidean Distance 95
2.3.3 Big Picture 100
2.4 Proxy methods 103
2.5 Advanced Methods 103
2.5.1 Statistical Distance 104
2.5.2 Structured Metric Learning 108
2.6 Application Gesture Sensing 110
2.6.1 Radar System Design 111
2.6.2 Data Set and Preparation 112
2.6.3 Architecture and Metric Learning Procedure 114
2.6.4 Results 123
2.7 Questions to the Reader 129
Bibliography 130
3 Deep Parametric Learning 135
3.1 Introduction 135
3.2 Radar Parametric Neural Network 140
3.2.1 2D Sinc Filters 142
3.2.2 2D Morlet Wavelets 143
3.2.3 Adaptive 2D Sinc Filters 145
3.2.4 Complex Frequency Extraction Layer 146
3.3 Multilevel Wavelet Decomposition Network 150
3.4 Application Activity Classification 153
3.4.1 Proposed Parametric Networks 155
3.4.2 State-of-art Networks 158
3.4.3 Results & Discussion 160
3.5 Conclusion 167
3.6 Question to Readers 168
Bibliography 168
4 Deep Reinforcement Learning 173
4.1 Useful Notation and Equations 173
4.1.1 Markov Decision Process 173
4.1.2 Solving the Markov Decision Process 174
4.1.3 Bellman Equations 175
4.2 Introduction 175
4.3 On-Policy Reinforcement Learning 179
4.4 Off-Policy Reinforcement Learning 180
4.5 Model-Based Reinforcement Learning 180
4.6 Model-Free Reinforcement Learning 181
4.7 Value-Based Reinforcement Learning 181
4.8 Policy-Based Reinforcement Learning 183
4.9 Online Reinforcement Learning 183
4.10 Offline Reinforcement Learning 184
4.11 Reinforcement Learning with
Discrete Actions 184
4.12 Reinforcement Learning with
Continuous Actions 185
4.13 Reinforcement Learning Algorithms
for Radar Applications 185
4.14 Application Tracker's Parameter Optimization 189
4.14.1 Motivation 190
4.14.2 Background 192
4.14.3 Approach 202
4.14.4 Experimental 208
4.14.5 Outcomes of the proposed Approach 219
4.15 Conclusion 220
4.16 Questions to the Reader 220
Bibliography 221
5 Cross-Modal Learning 229
5.1 Introduction 229
5.2 Self-Supervised Multi-Modal Learning 233
5.2.1 Generating Audio Statistics 233
5.2.2 Predicting sounds from images 234
5.2.3 Audio Features Clustering 234
5.2.4 Binary Coding Model 235
5.2.5 Training 235
5.2.6 Results 235
5.3 Joint Embeddings Learning 237
5.3.1 Feature Representations 237
5.3.2 Joint-Embedding Learning 238
5.3.3 Matching & Ranking 239
5.3.4 Training Details & Result 239
5.3.5 Discussion 241
5.4 Multi-Modal Input 241
5.4.1 Multi-modal Compact Bilinear Pooling 242
5.4.2 VQA Architecture 243
5.4.3 Training Details & Result 245
5.4.4 Discussion 245
5.5 Cross-Modal Learning 245
5.5.1 Data Acquisition 246
5.5.2 Cross-Modal Learning for Key-Point Detection 246
5.5.3 Training Details & Result 247
5.5.4 Discussion 249
5.6 Application People Counting 250
5.6.1 FMCW Radar System Design 251
5.6.2 Data Acquisition 252
5.6.3 Solution 1 253
5.6.4 Solution 2 262
5.7 Conclusion 265
5.8 Questions to the Reader 265
Bibliography 267
6 Signal Processing with Deep Learning 273
6.1 Introduction 273
6.2 Algorithm Unrolling 274
6.2.1 Learning Fast Approximations of Sparse Coding 275
6.2.2 Learned ISTA in radar processing 279
6.3 Physics-inspired Deep Learning 282
6.4 Processing-specific Network Architectures 284
6.5 Deep Learning-aided Signal Processing 288
6.6 Questions to the Reader 297
Bibliography 297
7 Domain Adaptation 303
7.1 Introduction 303
7.2 Transfer Learning and Domain Adaptaton 304
7.3 Categories of Domain Adaptation 307
7.3.1 Common Data Shifts 307
7.3.2 Methods of Domain Adaptation 308
7.4 Domain Adaptation in Radar Processing 315
7.4.1 Domain Adaptation with a different Sensor Type 316
7.4.2 Domain Adaptation with different Radar Settings 318
7.5 Summary 331
7.6 Questions to the Reader 331
Bibliography 332
8 Bayesian Deep Learning 339
8.1 Learning Theory 341
8.2 Bayesian Learning 343
8.3 Bayesian Approximations 352
8.4 Application VRU Classification 372
8.4.1 VAE as Bayesian 373
xiii
8.4.2 Bayesian Metric Learning 377
8.4.3 Kalman as Bayesian 383
8.4.4 Results 387
8.5 Summary 391
8.6 Questions to the Reader 393
Bibliography 393
9 Geometric Deep Learning 397
9.1 Representation Learning in Graph Neural Network 399
9.1.1 Fundamentals 399
9.1.2 Learning Theory 401
9.1.3 Embedding Learning 406
9.2 Graph Representation Learning 407
9.2.1 Convolution GNN 408
9.2.2 Recurrent Graph Neural Networks (RGNN) 409
9.2.3 Graph Autoencoders (GAE) 409
9.2.4 Spatial-Temporal Graph Neural Networks (STGNN) 410
9.2.5 Attention GNN 410
9.2.6 Message-passing GNN 411
9.3 Applications 413
9.3.1 Application 1 Long-Range Gesture Recognition 413
9.3.2 Application 2 Bayesian Anchor-Free Target Detection 426
9.4 Conclusion 444
9.5 Questions to the Reader 445
Bibliography 446
1
Introduction to Radar Processing and Deep Learning
At the end of this chapter, reader will have understanding on
- How radar data cubes are processed to extract range, velocity, and angle of the multiple detected targets and are tracked over time.
- Different target representations used for radar target recognition.
- Introduction to deep learning architectures used for radar target recognition.
1.1 Basics of Radar Systems
Radar is an acronym that stands for radio detection and ranging. It is basically an electromagnetic system used to detect the presence of one or more targets of interest and estimate their range, angle, and velocity relative to the radar. Instead of just measuring the target's location and velocity, modern radars can predict the target given the reflected radar signals. The main objective of radar compared to infrared and optical sensors is to discover distant targets under difficult climate conditions and to determine their spatial location while tracking them over time with precision. The general working principle and signal processing fundamental details are explained in the following sections.
1.1.1 Fundamentals
The radar system generally consists of a transmitter that produces an electromagnetic signal, which is radiated into space by the transmit antenna. When this signal strikes an object, it gets reflected or re-radiated in many directions. This reflected echo signal is received by the receive antenna, which delivers it to the receiver circuitry, where it is processed to detect the target and also localize it over time along with certain characteristics of the target. A simplified version of a typical continuous wave radar front-end with the most important building blocks can be seen in Figure 1.1. The chosen waveform is generated by a local oscillator (LO) and transmitted via the transmit (Tx) antenna. The receive (Rx) antenna then captures the incoming signal reflections from the target at a distance. After amplifying the received signal, it is mixed with the original transmitted waveform and is passed through subsequent analog bandpass filtering (BPF). This removes any high-frequency components that could cause aliasing as well as low-frequency components from direct coupling of the LO signal into the receiver. After mixing and filtering the signal that has been shifted to an intermediate frequency (IF), and it is referred as . The IF bandwidth is determined by the upper cut-off frequency of the bandpass filter, which is typically in the order of tens of kHz to few MHz.
Figure 1.1 Block diagram of the continuous wave radar front-end and its receive chain including the mixer, band pass filter, and analog-to-digital converter. The digitized samples are stored into a data matrix . The radar in this case is sensing a human target in the field of view.
1.1.2 Signal Modulation
To detect and differentiate multiple targets along its range, relative velocity, and azimuth-elevation angle dimensions, linear frequency modulated continuous wave (FMCW) is used as the most standard sensing waveform [1]. Usually, consecutive identical chirps are transmitted within a frame with a predefined time spacing referred to as chirp repetition time. The received IF signal is arranged within a two dimensional matrix, and the intratime, i.e., within a chirp, is referred as fast-time, while the intertime, i.e., across chirps, is referred to as slow-time. If the target is static, the round trip delay in the received signal is manifested as a frequency offset along the fast-time dimension after down-mixing at the receiver. But if the target or the radar is not stationary, the received signal will have an additional frequency offset caused by the Doppler shift manifested across slow-time dimension.
Figure 1.2 shows the concept of a FMCW modulation in detail. The LO generates a chirp signal with starting frequency , bandwidth , duration , and resulting sweep rate . By taking advantage of time integral over Tx frequency, instantaneous phase is calculated as shown in Eq. (1.1), where corresponds to the initial phase of the LO:
(1.1a) (1.1b)Figure 1.2 Illustration of typical modern radar sensors with several identical transmit chirps within a frame and the digitized IF samples are then stored chirpwise in a data matrix for coherent processing.
Assuming unity amplitude for a single chirp, can be formulated by
(1.2)If this transmit signal gets reflected by some object, also referred to as target, the reflection will be received at the radar with a time delay , which is proportional to the target's distance to the radar. Additionally, signals of multiple reflections, like extended targets, are superimposed on each other at the receiver. For an arbitrary number of point targets composing a spatially distributed target, the received signal can thus be expressed as follows:
(1.3)where represents thermal receiver noise or clutter and is the round trip delay to the th target located at distance and moving with a relative radial velocity of . As a result, can be described as , where is the speed of light. For ease of notations, the noise term is dropped for all the following considerations. The received and amplified signal is mixed with the original transmitted signal (). As discussed before, both transmitted and received signal follows cosine waveform. Thus, the down-mixed signal can be transformed into two components using trigonometric formulation.
(1.4)here contains the difference of Tx and Rx signal frequencies and contains the sum frequencies respectively. The sum component is removed by the following BPF and the resulting IF signal is obtained as follows:
(1.5)This shows that intermediate received signal contains both distance-dependent frequency and also speed-dependent frequency shift which are factors of modulation parameters. This includes chirp duration , chirp repetition time , sweep frequency or bandwidth , and number of chirps in a frame as main configuration parameter for the design of a FMCW waveform. As a result, these parameters control the range and Doppler resolution, as presented in Eqs. (1.6) and (1.7), respectively. The maximum observable range and the maximum unambiguous Doppler is given in Eqs. (1.8) and (1.9).
(1.6a) (1.6b) (1.7a) (1.7b) (1.8) (1.9)1.2 FMCW Signal Processing
The IF signal, obtained from Eq. (1.5), is then digitized by an analog-to-digital converter with sampling period at the discrete time instants , where . Consequently, the discrete time signal contains samples per chirp. Typically, modern short-range radar sensors rapidly transmit several identical chirps in a so-called chirp sequence modulation. The digitized IF samples are then stored chirp wise in a data matrix for coherent processing. Figure 1.3 summarizes the FMCW signal processing multistage pipeline, where is first preprocessed in time-domain for removal of spectral leakage or static targets followed by interference mitigation. Later, the preprocessed is transformed to frequency domain for target detection. Once a target is detected, then the measurement is fed into a tracking algorithm for temporal smoothening. At the end, the tracked target's features are extracted using their motion or spatial signature in the form of images or point-clouds, respectively, which are used for target recognition.
Figure 1.3 Summary of FMCW signal processing pipeline including both pre- and postprocessing over chirp matrix for target detection, tracking, and classification.
The frequency domain analysis for range-Doppler processing is explained in the following section.
1.2.1 Frequency-Domain Analysis
As indicated by Eqs. (1.6) and (1.7) both range and radial velocity information of the target are functions of frequency shifts in the received signal. As a result, frequency-domain analysis is used to determine respective target's parameters instead of time-domain analysis. In contrast to time-domain signals where signal changes over time (amplitude or power) can be observed, frequency-domain analysis reveals how much of the signal lies within each given frequency band over a range of frequencies, which also include change in phase information. The most common frequency-domain transform methods are Fourier transform, short-time Fourier transform (STFT) and wavelet transforms. All three transforms are inner products of a family of basis functions with a time-domain signal. The parameterization and the basis functions determine the properties of the transforms.
Before delving into details, Figure 1.4 illustrates all the three transforms pictorially. While the classical technique to represent time signals in the frequency domain by calculating discrete Fourier transformation (DFT), it fails to detect time variant frequency effects, which are important for extended targets. As an...
