Digital Signal Processing with Kernel Methods
Description
More details
Other editions
Additional editions
Persons
JOSÉ LUIS ROJO-ÁLVAREZ, PhD, is a Professor in the Department of Signal Theory and Communications at the University Rey Juan Carlos, Fuenlabrada (Madrid) and Center for Computational Simulation, Universidad Politécnica de Madrid, Spain.
MANEL MARTÍNEZ-RAMÓN, PhD, is a Professor in the Department of Electrical and Computer Engineering at the University of New Mexico, Albuquerque, USA.
JORDI MUÑOZ-MARÍ, PhD, is an Associate Professor in the Department of Electronics Engineering at the Universitat de València, Spain.
GUSTAU CAMPS-VALLS, PhD, is an Associate Professor in the Department of Electronics Engineering at the Universitat de València, Spain.
Content
About the Authors xiii
Preface xvii
Acknowledgements xxi
List of Abbreviations xxiii
Part I Fundamentals and Basic Elements 1
1 From Signal Processing to Machine Learning 3
1.1 A New Science is Born: Signal Processing 3
1.1.1 Signal Processing Before Being Coined 3
1.1.2 1948: Birth of the Information Age 4
1.1.3 1950s: Audio Engineering Catalyzes Signal Processing 4
1.2 From Analog to Digital Signal Processing 5
1.2.1 1960s: Digital Signal Processing Begins 5
1.2.2 1970s: Digital Signal Processing Becomes Popular 6
1.2.3 1980s: Silicon Meets Digital Signal Processing 6
1.3 Digital Signal Processing Meets Machine Learning 7
1.3.1 1990s: New Application Areas 7
1.3.2 1990s: Neural Networks, Fuzzy Logic, and Genetic Optimization 7
1.4 Recent Machine Learning in Digital Signal Processing 8
1.4.1 Traditional Signal Assumptions Are No Longer Valid 8
1.4.2 Encoding Prior Knowledge 8
1.4.3 Learning and Knowledge from Data 9
1.4.4 From Machine Learning to Digital Signal Processing 9
1.4.5 From Digital Signal Processing to Machine Learning 10
2 Introduction to Digital Signal Processing 13
2.1 Outline of the Signal Processing Field 13
2.1.1 Fundamentals on Signals and Systems 14
2.1.2 Digital Filtering 21
2.1.3 Spectral Analysis 24
2.1.4 Deconvolution 28
2.1.5 Interpolation 30
2.1.6 System Identification 31
2.1.7 Blind Source Separation 36
2.2.3 Sparsity, Compressed Sensing, and Dictionary Learning 44
2.3 Multidimensional Signals and Systems 48
2.3.1 Multidimensional Signals 49
2.3.2 Multidimensional Systems 51
2.4 Spectral Analysis on Manifolds 52
2.4.1 Theoretical Fundamentals 52
2.4.2 Laplacian Matrices 54
2.5 Tutorials and Application Examples 57
2.5.1 Real and Complex Signal Processing and Representations 57
2.5.2 Convolution, Fourier Transform, and Spectrum 63
2.5.3 Continuous-Time Signals and Systems 67
2.5.4 Filtering Cardiac Signals 70
2.5.5 Nonparametric Spectrum Estimation 74
2.5.6 Parametric Spectrum Estimation 77
2.5.7 Source Separation 81
2.5.8 Time-Frequency Representations and Wavelets 84
2.5.9 Examples for Spectral Analysis on Manifolds 87
2.6 Questions and Problems 94
3 Signal Processing Models 97
3.1 Introduction 97
3.2 Vector Spaces, Basis, and Signal Models 98
3.2.1 Basic Operations for Vectors 98
3.2.2 Vector Spaces 100
3.2.3 Hilbert Spaces 101
3.2.4 Signal Models 102
3.2.5 Complex Signal Models 104
3.2.6 Standard Noise Models in Digital Signal Processing 105
3.2.7 The Role of the Cost Function 107
3.2.8 The Role of the Regularizer 109
3.3 Digital Signal Processing Models 111
3.3.1 Sinusoidal Signal Models 112
3.3.2 System Identification Signal Models 113
3.3.3 Sinc Interpolation Models 116
3.3.4 Sparse Deconvolution 120
3.3.5 Array Processing 121
3.4 Tutorials and Application Examples 122
3.4.1 Examples of Noise Models 123
3.4.2 Autoregressive Exogenous System Identification Models 132
3.4.3 Nonlinear System Identification Using Volterra Models 138
3.4.4 Sinusoidal Signal Models 140
3.4.5 Sinc-based Interpolation 144
3.4.6 Sparse Deconvolution 152
3.4.7 Array Processing 157
3.5 Questions and Problems 160
3.A MATLABsimpleInterp Toolbox Structure 161
4 Kernel Functions and Reproducing Kernel Hilbert Spaces 165
4.1 Introduction 165
4.2 Kernel Functions and Mappings 169
4.2.1 Measuring Similarity with Kernels 169
4.2.2 Positive-De¿nite Kernels 169
4.2.3 Reproducing Kernel in Hilbert Space and Reproducing Property 170
4.2.4 Mercer's Theorem 173
4.3 Kernel Properties 174
4.3.1 Tikhonov's Regularization 175
4.3.2 Representer Theorem and Regularization Properties 176
4.3.3 Basic Operations with Kernels 178
4.4 Constructing Kernel Functions 179
4.4.1 Standard Kernels 179
4.4.2 Properties of Kernels 180
4.4.3 Engineering Signal Processing Kernels 181
4.5 Complex Reproducing Kernel in Hilbert Spaces 184
4.6 Support Vector Machine Elements for Regression and Estimation 186
4.6.1 Support Vector Regression Signal Model and Cost Function 186
4.6.2 Minimizing Functional 187
4.7 Tutorials and Application Examples 191
4.7.1 Kernel Calculations and Kernel Matrices 191
4.7.2 Basic Operations with Kernels 194
4.7.3 Constructing Kernels 197
4.7.4 Complex Kernels 199
4.7.5 Application Example for Support Vector Regression Elements 202
4.8 Concluding Remarks 205
4.9 Questions and Problems 205
Part II Function Approximation and Adaptive Filtering 209
5 A Support Vector Machine Signal Estimation Framework 211
5.1 Introduction 211
5.2 A Framework for Support Vector Machine Signal Estimation 213
5.3 Primal Signal Models for Support Vector Machine Signal Processing 216
5.3.1 Nonparametric Spectrum and System Identification 218
5.3.2 Orthogonal Frequency Division Multiplexing Digital Communications 220
5.3.3 Convolutional Signal Models 222
5.3.4 Array Processing 225
5.4 Tutorials and Application Examples 227
5.4.1 Nonparametric Spectral Analysis with Primal Signal Models 227
5.4.2 System Identification with Primal Signal Model ;;-¿lter 228
5.4.3 Parametric Spectral Density Estimation with Primal Signal Models 230
5.4.4 Temporal Reference Array Processing with Primal Signal Models 231
5.4.5 Sinc Interpolation with Primal Signal Models 233
6 Reproducing Kernel Hilbert Space Models for Signal Processing 241
6.1 Introduction 241
6.2 Reproducing Kernel Hilbert Space Signal Models 242
6.2.1 Kernel Autoregressive Exogenous Identi¿cation 244
6.2.2 Kernel Finite Impulse Response and the ;;-Filter 247
6.2.3 Kernel Array Processing with Spatial Reference 248
6.2.4 Kernel Semiparametric Regression 249
6.3 Tutorials and Application Examples 258
6.3.1 Nonlinear System Identification with Support Vector Machine-Autoregressive and Moving Average 258
6.3.2 Nonlinear System Identi¿cation with the ;;-¿lter 260
6.3.3 Electric Network Modeling with Semiparametric Regression 264
6.3.4 Promotional Data 272
6.3.5 Spatial and Temporal Antenna Array Kernel Processing 275
6.4 Questions and Problems 279
7 Dual Signal Models for Signal Processing 281
7.1 Introduction 281
7.2 Dual Signal Model Elements 281
7.3 Dual Signal Model Instantiations 283
7.3.1 Dual Signal Model for Nonuniform Signal Interpolation 283
7.3.2 Dual Signal Model for Sparse Signal Deconvolution 284
7.3.3 Spectrally Adapted Mercer Kernels 285
7.4 Tutorials and Application Examples 289
7.4.1 Nonuniform Interpolation with the Dual Signal Model 290
7.4.2 Sparse Deconvolution with the Dual Signal Model 292
7.4.3 Doppler Ultrasound Processing for Fault Detection 294
7.4.4 Spectrally Adapted Mercer Kernels 296
7.4.5 Interpolation of Heart Rate Variability Signals 304
7.4.6 Denoising in Cardiac Motion-Mode Doppler Ultrasound Images 309?m
7.4.7 Indoor Location from Mobile Devices Measurements 316
7.4.8 Electroanatomical Maps in Cardiac Navigation Systems 322
7.5 Questions and Problems 331
8 Advances in Kernel Regression and Function Approximation 333
8.1 Introduction 333
8.2 Kernel-Based Regression Methods 333
8.2.1 Advances in Support Vector Regression 334
8.2.2 Multi-output Support Vector Regression 338
8.2.3 Kernel Ridge Regression 339
8.2.4 Kernel Signal-To-Noise Regression 341
8.2.5 Semisupervised Support Vector Regression 343
8.2.6 Model Selection in Kernel Regression Methods 345
8.4.1 Comparing Support Vector Regression, Relevance Vector Machines, and Gaussian Process Regression 360
8.4.2 Pröle-Dependent Support Vector Regression 362
8.4.3 Multi-output Support Vector Regression 364
8.4.4 Kernel Signal-to-Noise Ratio Regression 366
8.4.5 Semisupervised Support Vector Regression 368
8.4.6 Bayesian Nonparametric Model 369
8.4.7 Gaussian Process Regression 370
8.4.8 Relevance Vector Machines 379
8.5 Concluding Remarks 382
8.6 Questions and Problems 383
9 Adaptive Kernel Learning for Signal Processing 387
9.1 Introduction 387
9.2 Linear Adaptive Filtering 387
9.2.1 Least Mean Squares Algorithm 388
9.2.2 Recursive Least-Squares Algorithm 389
9.3 Kernel Adaptive Filtering 392
9.4 Kernel Least Mean Squares 392
9.4.1 Derivation of Kernel Least Mean Squares 393
9.4.2 Implementation Challenges and Dual Formulation 394
9.5.3 Prediction of the Mackey-Glass Time Series with Kernel Recursive Least Squares 401
9.5.4 Beyond the Stationary Model 402
9.5.5 Example on Nonlinear Channel Identi¿cation and Reconvergence 405
9.6 Explicit Recursivity for Adaptive Kernel Models 406
9.6.1 Recursivity in Hilbert Spaces 406
9.6.2 Recursive Filters in Reproducing Kernel Hilbert Spaces 408
9.7 Online Sparsi¿cation with Kernels 411
9.7.1 Sparsity by Construction 411
9.7.2 Sparsity by Pruning 413
9.8 Probabilistic Approaches to Kernel Adaptive Filtering 414
9.8.1 Gaussian Processes and Kernel Ridge Regression 415
9.8.2 Online Recursive Solution for Gaussian Processes Regression 416
9.8.3 Kernel Recursive Least Squares Tracker 417
9.8.4 Probabilistic Kernel Least Mean Squares 418
9.9 Further Reading 418
9.9.1 Selection of Kernel Parameters 418
9.9.2 Multi-Kernel Adaptive Filtering 419
9.9.3 Recursive Filtering in Kernel Hilbert Spaces 419
9.10 Tutorials and Application Examples 419
9.10.1 Kernel Adaptive Filtering Toolbox 420
9.10.2 Prediction of a Respiratory Motion Time Series 421
9.10.3 Online Regression on the KIN?h?eK Dataset 423
9.10.4 The Mackey-Glass Time Series 425
9.10.5 Explicit Recursivity on Reproducing Kernel in Hilbert Space and Electroencephalogram Prediction 427
9.10.6 Adaptive Antenna Array Processing 428
9.11 Questions and Problems 430
Part III Classification, Detection, and Feature Extraction 433
10 Support Vector Machine and Kernel Classification Algorithms 435
10.1 Introduction 435
10.2 Support Vector Machine and Kernel Classi¿ers 435
10.2.1 Support Vector Machines 435
10.2.2 Multiclass and Multilabel Support Vector Machines 441
10.2.3 Least-Squares Support Vector Machine 447
10.2.4 Kernel Fisher's Discriminant Analysis 448
10.3 Advances in Kernel-Based Classi¿cation 452
10.3.1 Large Margin Filtering 452
10.3.2 Semisupervised Learning 454
10.3.3 Multiple Kernel Learning 460
10.3.4 Structured-Output Learning 462
10.3.5 Active Learning 468
10.4 Large-Scale Support Vector Machines 477
10.4.1 Large-Scale Support Vector Machine Implementations 477
10.4.2 Random Fourier Features 478
10.4.3 Parallel Support Vector Machine 480
10.4.4 Outlook 483
10.5 Tutorials and Application Examples 485
10.5.1 Examples of Support Vector Machine Classi¿cation 485
10.5.2 Example of Least-Squares Support Vector Machine 492
10.5.3 Kernel-Filtering Support Vector Machine for Brain-Computer Interface Signal Classi¿cation 493
10.5.4 Example of Laplacian Support Vector Machine 494
10.5.5 Example of Graph-Based Label Propagation 498
10.5.6 Examples of Multiple Kernel Learning 498
10.6 Concluding Remarks 501
10.7 Questions and Problems 502
11 Clustering and Anomaly Detection with Kernels 503
11.1 Introduction 503
11.2 Kernel Clustering 506
11.2.1 Kernelization of the Metric 506
11.2.2 Clustering in Feature Spaces 508
11.3 Domain Description Via Support Vectors 514
11.3.1 Support Vector Domain Description 514
11.3.2 One-Class Support Vector Machine 515
11.3.3 Relationship Between Support Vector Domain Description and Density Estimation 516
11.3.4 Semisupervised One-Class Classi¿cation 517
11.4 Kernel Matched Subspace Detectors 518
11.4.1 Kernel Orthogonal Subspace Projection 518
11.4.2 Kernel Spectral Angle Mapper 520
11.5 Kernel Anomaly Change Detection 522
11.5.1 Linear Anomaly Change Detection Algorithms 522
11.5.2 Kernel Anomaly Change Detection Algorithms 523
11.6 Hypothesis Testing with Kernels 525
11.6.1 Distribution Embeddings 526
11.6.3 Maximum Mean Discrepancy 527
11.6.3 One-Class Support Measure Machine 528
11.7 Tutorials and Application Examples 529
11.7.1 Example on Kernelization of the Metric 529
11.7.2 Example on Kernel k-Means 530
11.7.3 Domain Description Examples 531
11.7.4 Kernel Spectral Angle Mapper and Kernel Orthogonal Subspace Projection Examples 534
11.7.5 Example of Kernel Anomaly Change Detection Algorithms 536
11.7.6 Example on Distribution Embeddings and Maximum Mean Discrepancy 540
11.8 Concluding Remarks 541
11.9 Questions and Problems 542
12 Kernel Feature Extraction in Signal Processing 543
12.1 Introduction 543
12.2 Multivariate Analysis in Reproducing Kernel Hilbert Spaces 545
12.2.1 Problem Statement and Notation 545
12.2.2 Linear Multivariate Analysis 546
12.2.3 Kernel Multivariate Analysis 549
12.2.4 Multivariate Analysis Experiments 551
12.3 Feature Extraction with Kernel Dependence Estimates 555
12.3.1 Feature Extraction Using Hilbert-Schmidt Independence Criterion 556
12.3.2 Blind Source Separation Using Kernels 563
12.4 Extensions for Large-Scale and Semisupervised Problems 570
12.4.2 E¿ciency with the Incomplete Cholesky Decomposition 570
12.4.3 E¿ciency with Random Fourier Features 570
12.4.3 Sparse Kernel Feature Extraction 571
12.4.4 Semisupervised Kernel Feature Extraction 573
12.5 Domain Adaptation with Kernels 575
12.5.1 Kernel Mean Matching 578
12.5.2 Transfer Component Analysis 579
12.5.3 Kernel Manifold Alignment 581
12.5.4 Relations between Domain Adaptation Methods 585
12.5.5 Experimental Comparison between Domain Adaptation Methods
12.6 Concluding Remarks 587
12.7 Questions and Problems 588
References 589 Index 631
About the Authors
José Luis Rojo-Álvarez received the Telecommunication Engineering degree in 1996 from University of Vigo, Spain, and a PhD in Telecommunication Engineering in 2000 from the Polytechnic University of Madrid, Spain. Since 2016, he has been a full Professor in the Department of Signal Theory and Communications, University Rey Juan Carlos, Madrid, Spain. He has published more than 90 papers in indexed journals and more than 150 international conference communications. He has participated in more than 60 projects (with public and private fundings), and directed more than 10 of them, including several actions in the National Plan for Research and Fundamental Science. He was a senior researcher at the Prometeo program in Ecuador (Army University, 2013 to 2015) and research advisor at the Telecommunication Ministry. In 2016 he received the Rey Juan Carlos University Prize for Talented Researcher.
His main current research interests include statistical learning theory, digital signal processing, and complex system modeling, with applications to cardiac signals and image processing. Specifically, he is committed to the development of new electrocardiographic imaging systems, long-term cardiac monitoring intelligent systems, and big data for electronic recording and hospital information analysis at large scales.
Manel Martínez-Ramón received an MsD in Telecommunications Engineering from Universitat Politécnica de Catalunya in 1996, and a PhD in Communications Technologies from Universidad Carlos III de Madrid (Spain) in 1999. In 2004 he spent a 20-month postdoctoral period at the MIND Research Network (New Mexico, USA). He was an Associate Professor at Universidad Carlos III de Madrid until 2013. There, he held various positions from Associate Dean of the School of Engineering to Associate Vice-Chancellor for Infrastructures. He has taught more than 30 different undergraduate and graduate classes in different universities.
Since August 2013 he has been a full professor with the Department of Electrical and Computer Engineering at the University of New Mexico, where he was permanently appointed Prince of Asturias Endowed Chair of the University of New Mexico, later renamed to King Felipe VI Endowed Chair, which is sponsored by the Household of the King of Spain. He is head of the machine learning track of this department and he is the Associate Director of the Center of Emerging Energy Technologies of this university. He is currently a principal investigator of several projects funded by the National Science Foundation and other agencies.
He has co-authored more than 40 journal papers and about 60 conference papers, and several books and book chapters. His research interests are in applications of machine learning to cyberphysical systems, including first-responders systems, smart grids, and cognitive radio.
Jordi Muñoz-Marí was born in Valéncia, Spain, in 1970, and received a BSc degree in Physics (1993), a BSc degree in Electronics Engineering (1996), and a PhD degree in Electronics Engineering (2003) from the Universitat de Valéncia. He is currently an associate professor in the Electronics Engineering Department at the Universitat de Valéncia, where he teaches electronic circuits, digital signal processing, and data science. He is a research member of the Image and Signal Processing (ISP) group. His research activity is tied to the study and development of machine-learning algorithms for signal and image processing.
Gustau Camps-Valls received BSc degrees in Physics (1996) and in Electronics Engineering (1998) and a PhD degree in Physics (2002), all from the Universitat de Valéncia. He is currently an Associate Professor (hab. Full Professor) in the Department of Electronics Engineering. He is a research coordinator in the Image and Signal Processing (ISP) group. He is interested in the development of machine-learning algorithms for geoscience and remote-sensing data analysis. He is an author of 130 journal papers, more than 150 conference papers, 20 international book chapters, and editor of the books Kernel Methods in Bioengineering, Signal and Image Processing (IGI, 2007), Kernel Methods for Remote Sensing Data Analysis" (John Wiley & Sons, 2009), and Remote Sensing Image Processing (MC, 2011). He holds a Hirsch's index h = 47, entered the ISI list of Highly Cited Researchers in 2011, and Thomson Reuters ScienceWatch identified one of his papers on kernel-based analysis of hyperspectral images as a Fast Moving Front research. In 2015, he obtained the prestigious European Research Council (ERC) consolidator grant on Statistical Learning for Earth Observation Data Analysis. Since 2007 he has been a member of the Data Fusion Technical Committee of the IEEE GRSS, and since 2009 of the Machine Learning for Signal Processing Technical Committee of the IEEE SPS. He is a member of the MTG-IRS Science Team (MIST) of EUMETSAT. He is Associate Editor of the IEEE Transactions on Signal Processing, IEEE Signal Processing Letters, IEEE Geoscience and Remote Sensing Letters, and invited guest editor for IEEE Journal of Selected Topics in Signal Processing (2012) and IEEE Geoscience and Remote Sensing Magazine (2015).
Valero Laparra Pérez-Muelas received a BSc degree in Telecommunications Engineering (2005), a BSc degree in Electronics Engineering (2007), a BSc degree in Mathematics (2010), and a PhD degree in Computer Science and Mathematics (2011). Currently, he has a postdoctoral position in the Image Processing Laboratory (IPL) and an Assistant Professor position in the Department of Electronics Engineering at the Universitat de Valéncia.
Luca Martino obtained his PhD in Statistical Signal Processing from Universidad Carlos III de Madrid, Spain, in 2011. He has been an Assistant Professor in the Department of Signal Theory and Communications at Universidad Carlos III de Madrid since then. In August 2013 he joined the Department of Mathematics and Statistics at the University of Helsinki. In March 2015, he joined the Universidade de Sao Paulo (USP). Currently, he is a postdoctoral researcher at the Universitat de Valéncia. His research interests include Bayesian inference, Monte Carlo methods, and nonparametric regression techniques.
Sergio Muñoz-Romero earned his PhD in Machine Learning at Universidad Carlos III de Madrid, where he also received the Telecommunication Engineering degree. He has led pioneering projects where machine-learning knowledge was successfully used to solve real big-data problems. Currently, he is a researcher at Universidad Rey Juan Carlos. Since 2015, he has worked at Persei vivarium as Head of Data Science and Big Data. His present research interests are centered around machine-learning algorithms and statistical learning theory, mainly in dimensionality reduction and feature selection methods, and their applications to bioengineering and big data.
Adrián Pérez-Suay obtained his BSc degree in Mathematics (2007), a Master's degree in Advanced Computing and Intelligent Systems (2010), and a PhD degree in Computational Mathematics and Computer Science (2015) about distance metric learning, all from the Universitat de Valéncia. He is currently a postdoctoral researcher at the Image Processing Laboratory (IPL) working on feature extraction and classification problems in remote-sensing data analysis, and has worked as assistant professor in the Department of Mathematics at the Universitat de Valéncia.
Margarita Sanromán-Junquera received the Technical Telecommunication Engineering degree from Universidad Carlos III de Madrid, Spain, in 2007, the Telecommunication Engineering degree from Universidad Rey Juan Carlos, Spain, in 2009, an MSc in Biomedical Engineering and Telemedicine from the Universidad Politécnica de Madrid, Spain, in 2010, and a PhD in Multimedia and Communication from Universidad Rey Juan Carlos and Universidad Carlos III de Madrid, in 2014. She is currently an Assistant Professor in the Department of Signal Theory and Communications, Telematics, and Computing at Universidad Rey Juan Carlos. Her research interests include statistical learning theory, digital processing of images and signals, and their applications to bioengineering.
Cristina Soguero-Ruiz received the Telecommunication Engineering degree and a BSc degree in Business Administration and Management in 2011, and an MSc degree in Biomedical Engineering from the University Rey Juan Carlos, Madrid, Spain, in 2012. She obtained her PhD degree in Machine Learning with Applications in Healthcare in 2015 in the Joint Doctoral Program in Multimedia and Communications in conjunction with University Rey Juan Carlos and University Carlos III. She was supported by FPU Spanish Research and Teaching Fellowship (granted in 2012, third place in TEC area). She won the Orange Foundation Best PhD Thesis Award by the Spanish Official College of Telecommunication Engineering.
Steven Van-Vaerenbergh received his MSc degree in Electrical Engineering from Ghent University, Belgium, in 2003, and a PhD degree from the University of Cantabria, Santander, Spain, in 2010. He was a visiting researcher with the Computational Neuroengineering Laboratory, University of Florida, Gainesville, in 2008. Currently, he is a postdoctoral associate with the Department of Telecommunication Engineering, University of Cantabria, Spain, where he is the principal...
