Data-Variant Kernel Analysis
Published on 27. April 2015
256 pages
978-1-119-01934-3 (ISBN)
Description
Describes and discusses the variants of kernel analysis methods for data types that have been intensely studied in recent years
This book covers kernel analysis topics ranging from the fundamental theory of kernel functions to its applications. The book surveys the current status, popular trends, and developments in kernel analysis studies. The author discusses multiple kernel learning algorithms and how to choose the appropriate kernels during the learning phase. Data-Variant Kernel Analysis is a new pattern analysis framework for different types of data configurations. The chapters include data formations of offline, distributed, online, cloud, and longitudinal data, used for kernel analysis to classify and predict future state.
Data-Variant Kernel Analysis:
* Surveys the kernel analysis in the traditionally developed machine learning techniques, such as Neural Networks (NN), Support Vector Machines (SVM), and Principal Component Analysis (PCA)
* Develops group kernel analysis with the distributed databases to compare speed and memory usages
* Explores the possibility of real-time processes by synthesizing offline and online databases
* Applies the assembled databases to compare cloud computing environments
* Examines the prediction of longitudinal data with time-sequential configurations
Data-Variant Kernel Analysis is a detailed reference for graduate students as well as electrical and computer engineers interested in pattern analysis and its application in colon cancer detection.
More details
Other editions
Additional editions
Yuichi Motai
Data-Variant Kernel Analysis
Book
04/2015
1st Edition
Wiley
€135.50
Shipment within 15-20 days
Person
Yuichi Motai, Ph.D., is an Associate Professor of Electrical and Computer Engineering at the Virginia Commonwealth University, Richmond, Virginia. He received his Ph.D. with the Robot Vision Laboratory in the School of Electrical and Computer Engineering, Purdue University, West Lafayette, Indiana in 2002.
Content
List of Figures xiii
List of Tables xix
Preface xxiii
Acknowledgments xxv
Chapter 1 Survey 1
1.1 Introduction of Kernel Analysis 1
1.2 Kernel Offline Learning 2
1.2.1 Choose the Appropriate Kernels 3
1.2.2 Adopt KA into the Traditionally Developed Machine Learning Techniques 6
1.2.3 Structured Database with Kernel 9
1.3 Distributed Database with Kernel 12
1.3.1 Multiple Database Representation 12
1.3.2 Kernel Selections Among Heterogeneous Multiple Databases 13
1.3.3 Multiple Database Representation KA Applications to Distributed Databases 14
1.4 Kernel Online Learning 16
1.4.1 Kernel-Based Online Learning Algorithms 16
1.4.2 Adopt "Online" KA Framework into the Traditionally Developed Machine Learning Techniques 17
1.4.3 Relationship Between Online Learning and Prediction Techniques 21
1.5 Prediction with Kernels 22
1.5.1 Linear Prediction 22
1.5.2 Kalman Filter 23
1.5.3 Finite-State Model 23
1.5.4 Autoregressive Moving Average Model 24
1.5.5 Comparison of Four Models 25
1.6 Future Direction and Conclusion 26
References 26
Chapter 2 Offline Kernel Analysis 41
2.1 Introduction 41
2.2 Kernel Feature Analysis 43
2.2.1 Kernel Basics 43
2.2.2 Kernel Principal Component Analysis (KPCA) 45
2.2.3 Accelerated Kernel Feature Analysis (AKFA) 46
2.2.4 Comparison of the Relevant Kernel Methods 48
2.3 Principal Composite Kernel Feature Analysis (PC-KFA) 49
2.3.1 Kernel Selections 49
2.3.2 Kernel Combinatory Optimization 52
2.4 Experimental Analysis 54
2.4.1 Cancer Image Datasets 54
2.4.2 Kernel Selection 56
2.4.3 Kernel Combination and Reconstruction 58
2.4.4 Kernel Combination and Classification 59
2.4.5 Comparisons of Other Composite Kernel Learning Studies 60
2.4.6 Computation Time 61
2.5 Conclusion 62
References 63
Chapter 3 Group Kernel Feature Analysis 69
3.1 Introduction 69
3.2 Kernel Principal Component Analysis (KPCA) 71
3.3 Kernel Feature Analysis (KFA) for Distributed Databases 73
3.3.1 Extract Data-Dependent Kernels Using KFA 73
3.3.2 Decomposition of Database Through Data Association via Recursively Updating Kernel Matrices 75
3.4 Group Kernel Feature Analysis (GKFA) 78
3.4.1 Composite Kernel: Kernel Combinatory Optimization 79
3.4.2 Multiple Databases Using Composite Kernel 81
3.5 Experimental Results 83
3.5.1 Cancer Databases 83
3.5.2 Optimal Selection of Data-Dependent Kernels 84
3.5.3 Kernel Combinatory Optimization 84
3.5.4 Composite Kernel for Multiple Databases 86
3.5.5 K-NN Classification Evaluation with ROC 87
3.5.6 Comparison of Results with Other Studies on Colonography 89
3.5.7 Computational Speed and Scalability Evaluation of GKFA 90
3.6 Conclusions 91
References 92
Chapter 4 Online Kernel Analysis 97
4.1 Introduction 97
4.2 Kernel Basics: A Brief Review 99
4.2.1 Kernel Principal Component Analysis 99
4.2.2 Kernel Selection 100
4.3 Kernel Adaptation Analysis of PC-KFA 101
4.4 Heterogeneous vs. Homogeneous Data for Online PC-KFA 102
4.4.1 Updating the Gram Matrix of the Online Data 103
4.4.2 Composite Kernel for Online Data 104
4.5 Long-Term Sequential Trajectories with Self-Monitoring 104
4.5.1 Reevaluation of Large Online Data 105
4.5.2 Validation of Decomposing Online Data into Small Chunks 106
4.6 Experimental Results 107
4.6.1 Cancer Datasets 107
4.6.2 Selection of Optimum Kernel and Composite Kernel for Offline Data 108
4.6.3 Selection of Optimum Kernel and Composite Kernel for the New Online Sequences 110
4.6.4 Classification of Heterogeneous Versus Homogeneous Data 111
4.6.5 Online Learning Evaluation of Long-term Sequence 112
4.6.6 Evaluation of Computational Time 116
4.7 Conclusions 117
References 117
Chapter 5 Cloud Kernel Analysis 121
5.1 Introduction 121
5.2 Cloud Environments 123
5.2.1 Server Specifications of Cloud Platforms 123
5.2.2 Cloud Framework of KPCA for AMD 124
5.3 AMD for Cloud Colonography 125
5.3.1 AMD Concept 125
5.3.2 Data Configuration of AMD 125
5.3.3 Implementation of AMD for Two Cloud Cases 129
5.3.4 Parallelization of AMD 132
5.4 Classification Evaluation of Cloud Colonography 135
5.4.1 Databases with Classification Criteria 135
5.4.2 Classification Results 137
5.5 Cloud Computing Performance 140
5.5.1 Cloud Computing Setting with Cancer Databases 140
5.5.2 Computation Time 142
5.5.3 Memory Usage 144
5.5.4 Running Cost 145
5.5.5 Parallelization 145
5.6 Conclusions 146
References 147
Chapter 6 Predictive Kernel Analysis 153
6.1 Introduction 153
6.2 Kernel Basics 154
6.2.1 KPCA and AKFA 155
6.3 Stationary Data Training 157
6.3.1 Kernel Selection 157
6.3.2 Composite Kernel: Kernel Combinatory Optimization 159
6.4 Longitudinal Nonstationary Data with Anomaly/Normal Detection 160
6.4.1 Updating the Gram Matrix Based on Nonstationary Longitudinal Data 160
6.4.2 Composite Kernel for Nonstationary Data 162
6.5 Longitudinal Sequential Trajectories for Anomaly Detection and Prediction 163
6.5.1 Anomaly Detection of Nonstationary Small Chunks Datasets 164
6.5.2 Anomaly Prediction of Long-Time Sequential Trajectories 167
6.6 Classification Results 169
6.6.1 Cancer Datasets 169
6.6.2 Selection of Optimum Kernel and Composite Kernel for Stationary Data 170
6.6.3 Comparisons with Other Kernel Learning Methods 172
6.6.4 Anomaly Detection for the Nonstationary Data 174
6.7 Longitudinal Prediction Results 175
6.7.1 Large Nonstationary Sequential dataset for Anomaly Detection 175
6.7.2 Time Horizontal Prediction for Risk Factor Analysis of Anomaly Long-Time Sequential Trajectories 178
6.7.3 Computational Time for Complexity Evaluation 179
6.8 Conclusions 180
References 181
Chapter 7 Conclusion 185
Appendix A 189
Appendix B Representative Matlab codes 195
B.1 Accelerated Kernel Feature Analysis 196
B.2 Experimental Evaluations 198
B.3 Group Kernel Analysis 201
B.4 Online Composite Kernel Analysis 206
B.5 Online Data Sequences Control 208
B.6 Alignment Factor 209
B.7 Cloud Kernel Analysis 210
B.8 Plot Computation Time 211
B.9 Parallelization 212
Index 215
Figure 1.1 General RBF-NN [97]. The input and output layers perform functions similarly to a normal neural network. The hidden layer uses the RBF kernel, denoted as its activation function as shown in Equation 1.4. Figure 1.2 Principle of kernel PCA. (a) nonlinear in input space. (b) high-dimensional feature space corresponding to nonlinear input space. The contour lines of constant projections generate the principal eigenvectors. Kernel PCA does not actually require F to be mapped into F. All necessary computations are executed through the use of a kernel function k in input space R2. Figure 1.3 Significant research publications in various fields of KA and kernel selection for offline learning. Works related to neural network, SVM, PCA, and kernel selection are displayed according to citation number versus publishing year. Figure 1.4 The composition of the multiple databases. The multiple databases include distributed databases and single databases, which are stored on different storage spaces connected within a network. Three key attributes of the multiple databases are (i) simultaneous access and modification, (ii) extraction from huge amounts, and (iii) distributed clustering/classification. Figure 1.5 Several pregenerated kernels are trained to access different features in several distributed databases. The output of this kernel network is a sum of the kernel functions that maps to the data of interest. The summation and biasing make this method behave similarly to a neural network [178]. Figure 1.6 Online kernel-based learning algorithm. As a new training sample is added, the kernel is updated on the basis of the online learning algorithms described in the following section. Figure 1.7 Key papers for Kernel Online learning based on the number of citations for the years 1998-2013, for principal component analysis (PCA), support vector machine (SVM), neural network (NN), autoregressive moving average (ARMA), and finite-state model (FSM). Figure 1.8 Online NN learning algorithm. An artificial neural network consists of input, hidden, and output layers interconnected with directed weights (w), where Wij denotes the input-to-hidden layer weights at the hidden neuron j and i>wjk denotes the hidden-to-output layer weights at the output neuron k [218]. Figure 1.9 Scheme for naïve online SVM classification. The input data are mapped to the kernel Hilbert space, and the hyperplane is generated as in offline SVM. If an optimal solution cannot be found, a new kernel is selected. The process repeats for newly incoming data. Figure 1.10 Online KPCA diagram. In the initial (offline) training phase, the initial kernel parameters and eigen-feature space are calculated. As new training sets are introduced, incremental (online) adjustments are made to the parameters and an updated eigen-space is generated. Figure 1.11 Linear predictor with an initial kernel mapping. The input data is mapped from its original space to a new Hilbert space where x(n) is linear, by the function F to allow for the use of the kernel trick. The model is optimized by minimized the function e(N), which is the difference in the current and previous predicted value. Figure 1.12 For processing a nonlinear time series, a kernelized Kalman filter is an optimal way to execute the Kalman filter. With a nonlinear series and an associated series . is mapped to by the nonlinear map , where is the input space, H is the feature space, and . Figure 1.13 A kernel autoregressive moving average (KARMA) model with online learning for tracking of hand movement. Figure 2.1 The computation time comparison between KPCA, SKFA, AKFA, and PC-KFA as the data size increases. Figure 3.1 Distributed colonography with distributed image databases for colon cancer diagnosis. The hosting server collects and analyzes databases from different institutions and groups them into assembled databases. Figure 3.2 The concept of group kernel feature analysis. The proposed criteria are to determine the nature of database by (i) decomposition, (ii) classification by heterogeneity, and (iii) combination. Figure 3.3 Relationship of the criteria to determine homogenous and heterogeneous degree. Figure 3.4 The steps to choose the basic kernel matrix P ´ and the updating process according to the elements rf. Figure 3.5 The training flow chart of reclustered databases due to the heterogeneous nature. Figure 3.6 Group kernel feature analysis (GKFA). The first two steps 1 and 2 are the same as in Fig. 3.2. Step 3 of Fig. 3.2 is illustrated here for assembled databases through kernel choice in the composite kernel by Sections 3.4.1 and 3.4.2. Figure 3.7 The overall GKFA steps of newly assembled databases. Figure 3.8 The computation time comparison between KPCA, KFA, and GKFA as the data size increases. Figure 4.1 Developed CAD for CTC at 3D imaging group at Harvard Medical School [5]. Figure 4.2 Criteria concept for homogeneous and heterogeneous online data. Figure 4.3 Overall flow of online data training using HBDA with PC-KFA. Figure 4.4 Online decomposition for heterogeneous sequences. Figure 4.5 Training of online datasets acquired over long-term sequences. Figure 4.6 The ROC performance of offline data using PC-KFA. Figure 4.7 The online dataset sequences of Table 4.4. Figure 4.8 Long-term data sequence used to evaluate online learning. The horizontal axis denotes the ratio of number of online data to number of offline data. The vertical axis denotes the number of total data processed using online HBDA with PC-KFA. The three long sequences are labeled as "Large Online Data," "Medium Online Data," and "Small Online Data," which were used corresponding to offline training dataset size of 750, 937, and 1250, respectively. Figure 4.9 Alignment factors for long-term sequences. The horizontal axis denotes the progress of the online HBDA (PC-KFA) with time (ratio of online and offline training). The solid lines denoted the mean value of the observed AF value, while the dashed lines show the range of observed AF. Figure 4.10 AUC of Receiver operating Characteristics for three long-term sequences. The horizontal axis denotes the progress of the online HBDA (PC-KFA) with time (ratio of online-to-offline training). Figure 4.11 Accuracy versus ratio of online data to offline data. Figure 4.12 Computational time for online HBDA with offline PC-KFA. Figure 5.1 A concept of Cloud Colonography with distributed image databases for colon cancer diagnosis. The cloud server hosting will collect distributed databases from different institutions and group them using KA. Figure 5.2 An illustration of KPCA mathematical background. KPCA calculates the eigenvectors and eigenvalues by analyzing the kernel feature space of multiple institutions so that a cloud server can handle larger datasets. Figure 5.3 A concept of AMD. The proposed kernel framework combines the Cloud Colonography datasets by analyzing the images of polyps with nonlinear big feature space. Figure 5.4 Four representative steps of AMD. The proposed AMD consists of the four main criteria to manage databases by (i) split, (ii) combine, (iii) sort, and (iv) merge. Figure 5.5 Cloud Colonography architecture in private cloud environment. The proposed Cloud Colonography consists of the four representative layers from CTC CAD analysis to the service desk reporting for clinical users. Figure 5.6 An extension of AMD framework in public cloud scenario. Figure 5.7 A proposed parallelization organization for cloud computing. Figure 5.8 ROC with Sigmoid and Gauss group kernel. (a) Comparison between assembling...
LIST OF FIGURES
Figure 1.1 General RBF-NN [97]. The input and output layers perform functions similarly to a normal neural network. The hidden layer uses the RBF kernel, denoted as its activation function as shown in Equation 1.4. Figure 1.2 Principle of kernel PCA. (a) nonlinear in input space. (b) high-dimensional feature space corresponding to nonlinear input space. The contour lines of constant projections generate the principal eigenvectors. Kernel PCA does not actually require F to be mapped into F. All necessary computations are executed through the use of a kernel function k in input space R2. Figure 1.3 Significant research publications in various fields of KA and kernel selection for offline learning. Works related to neural network, SVM, PCA, and kernel selection are displayed according to citation number versus publishing year. Figure 1.4 The composition of the multiple databases. The multiple databases include distributed databases and single databases, which are stored on different storage spaces connected within a network. Three key attributes of the multiple databases are (i) simultaneous access and modification, (ii) extraction from huge amounts, and (iii) distributed clustering/classification. Figure 1.5 Several pregenerated kernels are trained to access different features in several distributed databases. The output of this kernel network is a sum of the kernel functions that maps to the data of interest. The summation and biasing make this method behave similarly to a neural network [178]. Figure 1.6 Online kernel-based learning algorithm. As a new training sample is added, the kernel is updated on the basis of the online learning algorithms described in the following section. Figure 1.7 Key papers for Kernel Online learning based on the number of citations for the years 1998-2013, for principal component analysis (PCA), support vector machine (SVM), neural network (NN), autoregressive moving average (ARMA), and finite-state model (FSM). Figure 1.8 Online NN learning algorithm. An artificial neural network consists of input, hidden, and output layers interconnected with directed weights (w), where Wij denotes the input-to-hidden layer weights at the hidden neuron j and i>wjk denotes the hidden-to-output layer weights at the output neuron k [218]. Figure 1.9 Scheme for naïve online SVM classification. The input data are mapped to the kernel Hilbert space, and the hyperplane is generated as in offline SVM. If an optimal solution cannot be found, a new kernel is selected. The process repeats for newly incoming data. Figure 1.10 Online KPCA diagram. In the initial (offline) training phase, the initial kernel parameters and eigen-feature space are calculated. As new training sets are introduced, incremental (online) adjustments are made to the parameters and an updated eigen-space is generated. Figure 1.11 Linear predictor with an initial kernel mapping. The input data is mapped from its original space to a new Hilbert space where x(n) is linear, by the function F to allow for the use of the kernel trick. The model is optimized by minimized the function e(N), which is the difference in the current and previous predicted value. Figure 1.12 For processing a nonlinear time series, a kernelized Kalman filter is an optimal way to execute the Kalman filter. With a nonlinear series and an associated series . is mapped to by the nonlinear map , where is the input space, H is the feature space, and . Figure 1.13 A kernel autoregressive moving average (KARMA) model with online learning for tracking of hand movement. Figure 2.1 The computation time comparison between KPCA, SKFA, AKFA, and PC-KFA as the data size increases. Figure 3.1 Distributed colonography with distributed image databases for colon cancer diagnosis. The hosting server collects and analyzes databases from different institutions and groups them into assembled databases. Figure 3.2 The concept of group kernel feature analysis. The proposed criteria are to determine the nature of database by (i) decomposition, (ii) classification by heterogeneity, and (iii) combination. Figure 3.3 Relationship of the criteria to determine homogenous and heterogeneous degree. Figure 3.4 The steps to choose the basic kernel matrix P ´ and the updating process according to the elements rf. Figure 3.5 The training flow chart of reclustered databases due to the heterogeneous nature. Figure 3.6 Group kernel feature analysis (GKFA). The first two steps 1 and 2 are the same as in Fig. 3.2. Step 3 of Fig. 3.2 is illustrated here for assembled databases through kernel choice in the composite kernel by Sections 3.4.1 and 3.4.2. Figure 3.7 The overall GKFA steps of newly assembled databases. Figure 3.8 The computation time comparison between KPCA, KFA, and GKFA as the data size increases. Figure 4.1 Developed CAD for CTC at 3D imaging group at Harvard Medical School [5]. Figure 4.2 Criteria concept for homogeneous and heterogeneous online data. Figure 4.3 Overall flow of online data training using HBDA with PC-KFA. Figure 4.4 Online decomposition for heterogeneous sequences. Figure 4.5 Training of online datasets acquired over long-term sequences. Figure 4.6 The ROC performance of offline data using PC-KFA. Figure 4.7 The online dataset sequences of Table 4.4. Figure 4.8 Long-term data sequence used to evaluate online learning. The horizontal axis denotes the ratio of number of online data to number of offline data. The vertical axis denotes the number of total data processed using online HBDA with PC-KFA. The three long sequences are labeled as "Large Online Data," "Medium Online Data," and "Small Online Data," which were used corresponding to offline training dataset size of 750, 937, and 1250, respectively. Figure 4.9 Alignment factors for long-term sequences. The horizontal axis denotes the progress of the online HBDA (PC-KFA) with time (ratio of online and offline training). The solid lines denoted the mean value of the observed AF value, while the dashed lines show the range of observed AF. Figure 4.10 AUC of Receiver operating Characteristics for three long-term sequences. The horizontal axis denotes the progress of the online HBDA (PC-KFA) with time (ratio of online-to-offline training). Figure 4.11 Accuracy versus ratio of online data to offline data. Figure 4.12 Computational time for online HBDA with offline PC-KFA. Figure 5.1 A concept of Cloud Colonography with distributed image databases for colon cancer diagnosis. The cloud server hosting will collect distributed databases from different institutions and group them using KA. Figure 5.2 An illustration of KPCA mathematical background. KPCA calculates the eigenvectors and eigenvalues by analyzing the kernel feature space of multiple institutions so that a cloud server can handle larger datasets. Figure 5.3 A concept of AMD. The proposed kernel framework combines the Cloud Colonography datasets by analyzing the images of polyps with nonlinear big feature space. Figure 5.4 Four representative steps of AMD. The proposed AMD consists of the four main criteria to manage databases by (i) split, (ii) combine, (iii) sort, and (iv) merge. Figure 5.5 Cloud Colonography architecture in private cloud environment. The proposed Cloud Colonography consists of the four representative layers from CTC CAD analysis to the service desk reporting for clinical users. Figure 5.6 An extension of AMD framework in public cloud scenario. Figure 5.7 A proposed parallelization organization for cloud computing. Figure 5.8 ROC with Sigmoid and Gauss group kernel. (a) Comparison between assembling...
