Theory of Ridge Regression Estimation with Applications
Description
Theory of Ridge Regression Estimation with Applications offers a comprehensive guide to the theory and methods of estimation. Ridge regression and LASSO are at the center of all penalty estimators in a range of standard models that are used in many applied statistical analyses. Written by noted experts in the field, the book contains a thorough introduction to penalty and shrinkage estimation and explores the role that ridge, LASSO, and logistic regression play in the computer intensive area of neural network and big data analysis.
Designed to be accessible, the book presents detailed coverage of the basic terminology related to various models such as the location and simple linear models, normal and rank theory-based ridge, LASSO, preliminary test and Stein-type estimators.The authors also include problem sets to enhance learning. This book is a volume in the Wiley Series in Probability and Statistics series that provides essential and invaluable reading for all statisticians. This important resource:
* Offers theoretical coverage and computer-intensive applications of the procedures presented
* Contains solutions and alternate methods for prediction accuracy and selecting model procedures
* Presents the first book to focus on ridge regression and unifies past research with current methodology
* Uses R throughout the text and includes a companion website containing convenient data sets
Written for graduate students, practitioners, and researchers in various fields of science, Theory of Ridge Regression Estimation with Applications is an authoritative guide to the theory and methodology of statistical estimation.
More details
Other editions
Additional editions
Persons
A. K. Md. EHSANES SALEH, PhD, is a Professor Emeritus and Distinguished Research Professor in the school of Mathematics and Statistics, Carleton University, Ottawa, Canada.
MOHAMMAD ARASHI, PhD, is an Associate Professor at Shahrood University of Technology, Iran and Extraordinary Professor and C2 rated researcher at University of Pretoria, Pretoria, South Africa.
B. M. GOLAM KIBRIA, PhD, is a Professor in the Department of Mathematics and Statistics at Florida International University, Miami, FL.
Content
List of Figures xvii
List of Tables xxi
Preface xxvii
Abbreviations and Acronyms xxxi
List of Symbols xxxiii
1 Introduction to Ridge Regression 1
1.1 Introduction 1
1.1.1 Multicollinearity Problem 3
1.2 Ridge Regression Estimator: Ridge Notion 5
1.3 LSE vs. RRE 6
1.4 Estimation of Ridge Parameter 7
1.5 Preliminary Test and Stein-Type Ridge Estimators 8
1.6 High-Dimensional Setting 9
1.7 Notes and References 11
1.8 Organization of the Book 12
2 Location and Simple Linear Models 15
2.1 Introduction 15
2.2 Location Model 16
2.2.1 Location Model: Estimation 16
2.2.2 Shrinkage Estimation of Location 17
2.2.3 Ridge Regression-Type Estimation of Location Parameter 18
2.2.4 LASSO for Location Parameter 18
2.2.5 Bias and MSE Expression for the LASSO of Location Parameter 19
2.2.6 Preliminary Test Estimator, Bias, and MSE 23
2.2.7 Stein-Type Estimation of Location Parameter 24
2.2.8 Comparison of LSE, PTE, Ridge, SE, and LASSO 24
2.3 Simple Linear Model 26
2.3.1 Estimation of the Intercept and Slope Parameters 26
2.3.2 Test for Slope Parameter 27
2.3.3 PTE of the Intercept and Slope Parameters 27
2.3.4 Comparison of Bias and MSE Functions 29
2.3.5 Alternative PTE 31
2.3.6 Optimum Level of Significance of Preliminary Test 33
2.3.7 Ridge-Type Estimation of Intercept and Slope 34
2.3.7.1 Bias and MSE Expressions 35
2.3.8 LASSO Estimation of Intercept and Slope 36
2.4 Summary and Concluding Remarks 39
3 ANOVA Model 43
3.1 Introduction 43
3.2 Model, Estimation, and Tests 44
3.2.1 Estimation of Treatment Effects 45
3.2.2 Test of Significance 45
3.2.3 Penalty Estimators 46
3.2.4 Preliminary Test and Stein-Type Estimators 47
3.3 Bias and Weighted L2 Risks of Estimators 48
3.3.1 Hard Threshold Estimator (Subset Selection Rule) 48
3.3.2 LASSO Estimator 49
3.3.3 Ridge Regression Estimator 51
3.4 Comparison of Estimators 52
3.4.1 Comparison of LSE with RLSE 52
3.4.2 Comparison of LSE with PTE 52
3.4.3 Comparison of LSE with SE and PRSE 53
3.4.4 Comparison of LSE and RLSE with RRE 54
3.4.5 Comparison of RRE with PTE, SE, and PRSE 56
3.4.5.1 Comparison Between ¿¿¿¿n^RR (kopt) and ¿¿¿¿n^PT (¿¿¿¿) 56
3.4.5.2 Comparison Between ¿¿¿¿n^RR (kopt) and ¿¿¿¿n^ s 56
3.4.5.3 Comparison of ¿¿¿¿n^RR (kopt) with ¿¿¿¿n^S+ 57
3.4.6 Comparison of LASSO with LSE and RLSE 58
3.4.7 Comparison of LASSO with PTE, SE, and PRSE 59
3.4.8 Comparison of LASSO with RRE 60
3.5 Application 60
3.6 Efficiency in Terms of Unweighted L2 Risk 63
3.7 Summary and Concluding Remarks 72
3A. Appendix 74
4 Seemingly Unrelated Simple Linear Models 79
4.1 Model, Estimation, and Test of Hypothesis 79
4.1.1 LSE of ¿¿¿¿ and ¿¿¿¿ 80
4.1.2 Penalty Estimation of ¿¿¿¿ and ¿¿¿¿ 80
4.1.3 PTE and Stein-Type Estimators of ¿¿¿¿ and ¿¿¿¿ 81
4.2 Bias and MSE Expressions of the Estimators 82
4.3 Comparison of Estimators 86
4.3.1 Comparison of LSE with RLSE 86
4.3.2 Comparison of LSE with PTE 86
4.3.3 Comparison of LSE with SE and PRSE 87
4.3.4 Comparison of LSE and RLSE with RRE 87
4.3.5 Comparison of RRE with PTE, SE, and PRSE 89
4.3.5.1 Comparison Between ¿¿¿¿n^RR (kopt) and ¿¿¿¿n^PT 89
4.3.5.2 Comparison Between ¿¿¿¿n^RR (kopt) and ¿¿¿¿n^S 89
4.3.5.3 Comparison of ¿¿¿¿n^RR (kopt) with ¿¿¿¿n^S+ 90
4.3.6 Comparison of LASSO with RRE 90
4.3.7 Comparison of LASSO with LSE and RLSE 92
4.3.8 Comparison of LASSO with PTE, SE, and PRSE 92
4.4 Efficiency in Terms of Unweighted L2 Risk 93
4.4.1 Efficiency for ¿¿¿¿ 94
4.4.2 Efficiency for ¿¿¿¿ 95
4.5 Summary and Concluding Remarks 96
5 Multiple Linear Regression Models 109
5.1 Introduction 109
5.2 Linear Model and the Estimators 110
5.2.1 Penalty Estimators 111
5.2.2 Shrinkage Estimators 113
5.3 Bias and Weighted L2 Risks of Estimators 114
5.3.1 Hard Threshold Estimator 114
5.3.2 Modified LASSO 116
5.3.3 Multivariate Normal Decision Theory and Oracles for Diagonal Linear Projection 117
5.3.4 Ridge Regression Estimator 119
5.3.5 Shrinkage Estimators 119
5.4 Comparison of Estimators 120
5.4.1 Comparison of LSE with RLSE 120
5.4.2 Comparison of LSE with PTE 121
5.4.3 Comparison of LSE with SE and PRSE 121
5.4.4 Comparison of LSE and RLSE with RRE 122
5.4.5 Comparison of RRE with PTE, SE, and PRSE 123
5.4.5.1 Comparison Between ¿¿¿¿n^RR (kopt) and ¿¿¿¿n^PT(¿¿¿¿) 123
5.4.5.2 Comparison Between ¿¿¿¿n^RR (kopt) and ¿¿¿¿n^S 124
5.4.5.3 Comparison of ¿¿¿¿n^RR (kopt) with ¿¿¿¿n^S+ 124
5.4.6 Comparison of MLASSO with LSE and RLSE 125
5.4.7 Comparison of MLASSO with PTE, SE, and PRSE 126
5.4.8 Comparison of MLASSO with RRE 127
5.5 Efficiency in Terms of Unweighted L2 Risk 127
5.6 Summary and Concluding Remarks 129
6 Ridge Regression in Theory and Applications 143
6.1 Multiple Linear Model Specification 143
6.1.1 Estimation of Regression Parameters 143
6.1.2 Test of Hypothesis for the Coefficients Vector 145
6.2 Ridge Regression Estimators (RREs) 146
6.3 Bias, MSE, and L2 Risk of Ridge Regression Estimator 147
6.4 Determination of the Tuning Parameters 151
6.5 Ridge Trace 151
6.6 Degrees of Freedom of RRE 154
6.7 Generalized Ridge Regression Estimators 155
6.8 LASSO and Adaptive Ridge Regression Estimators 156
6.9 Optimization Algorithm 158
6.9.1 Prostate Cancer Data 160
6.10 Estimation of Regression Parameters for Low-Dimensional Models 161
6.10.1 BLUE and Ridge Regression Estimators 161
6.10.2 Bias and L2-risk Expressions of Estimators 162
6.10.3 Comparison of the Estimators 165
6.10.4 Asymptotic Results of RRE 166
6.11 Summary and Concluding Remarks 168
7 Partially Linear Regression Models 171
7.1 Introduction 171
7.2 Partial Linear Model and Estimation 172
7.3 Ridge Estimators of Regression Parameter 174
7.4 Biases and L2 Risks of Shrinkage Estimators 177
7.5 Numerical Analysis 178
7.5.1 Example: Housing Prices Data 182
7.6 High-Dimensional PLM 188
7.6.1 Example: Riboflavin Data 192
7.7 Summary and Concluding Remarks 193
8 Logistic Regression Model 197
8.1 Introduction 197
8.1.1 Penalty Estimators 199
8.1.2 Shrinkage Estimators 200
8.1.3 Results on MLASSO 201
8.1.4 Results on PTE and Stein-Type Estimators 202
8.1.5 Results on Penalty Estimators 204
8.2 Asymptotic Distributional L2 Risk Efficiency Expressions of the Estimators 204
8.2.1 MLASSO vs. MLE 205
8.2.2 MLASSO vs. RMLE 206
8.2.3 Comparison of MLASSO vs. PTE 206
8.2.4 PT and MLE 207
8.2.5 Comparison of MLASSO vs. SE 208
8.2.6 Comparison of MLASSO vs. PRSE 208
8.2.7 RRE vs. MLE 209
8.2.7.1 RRE vs. RMLE 209
8.2.8 Comparison of RRE vs. PTE 211
8.2.9 Comparison of RRE vs. SE 211
8.2.10 Comparison of RRE vs. PRSE 212
8.2.11 PTE vs. SE and PRSE 212
8.2.12 Numerical Comparison Among the Estimators 213
8.3 Summary and Concluding Remarks 213
9 Regression Models with Autoregressive Errors 221
9.1 Introduction 221
9.1.1 Penalty Estimators 223
9.1.2 Shrinkage Estimators 224
9.1.2.1 Preliminary Test Estimator 224
9.1.2.2 Stein-Type and Positive-Rule Stein-Type Estimators 225
9.1.3 Results on Penalty Estimators 225
9.1.4 Results on PTE and Stein-Type Estimators 226
9.1.5 Results on Penalty Estimators 229
9.2 Asymptotic Distributional L2-risk Efficiency Comparison 230
9.2.1 Comparison of GLSE with RGLSE 230
9.2.2 Comparison of GLSE with PTE 231
9.2.3 Comparison of LSE with SE and PRSE 231
9.2.4 Comparison of LSE and RLSE with RRE 232
9.2.5 Comparison of RRE with PTE, SE and PRSE 233
9.2.5.1 Comparison Between ¿¿¿¿n^GRR(kopt) and ¿¿¿¿n^G(PT)233
9.2.5.2 Comparison Between ¿¿¿¿n^GRR(kopt) and ¿¿¿¿n^G(S) 234
9.2.5.3 Comparison of ¿¿¿¿n^GRR(kopt) with ¿¿¿¿n^G(S+) 234
9.2.6 Comparison of MLASSO with GLSE and RGLSE 235
9.2.7 Comparison of MLASSO with PTE, SE, and PRSE 236
9.2.8 Comparison of MLASSO with RRE 236
9.3 Example: Sea Level Rise at KeyWest, Florida 237
9.3.1 Estimation of the Model Parameters 237
9.3.1.1 Testing for Multicollinearity 237
9.3.1.2 Testing for Autoregressive Process 238
9.3.1.3 Estimation of Ridge Parameter k 239
9.3.2 Relative Efficiency 240
9.3.2.1 Relative Efficiency (REff) 240
9.3.2.2 Effect of Autocorrelation Coefficient ¿¿¿¿ 243
9.4 Summary and Concluding Remarks 245
10 Rank-Based Shrinkage Estimation 251
10.1 Introduction 251
10.2 LinearModel and Rank Estimation 252
10.2.1 Penalty R-Estimators 256
10.2.2 PTREs and Stein-type R-Estimators 258
10.3 Asymptotic Distributional Bias and L2 Risk of the R-Estimators 259
10.3.1 HardThreshold Estimators (Subset Selection) 259
10.3.2 Rank-based LASSO 260
10.3.3 Multivariate Normal DecisionTheory and Oracles for Diagonal Linear Projection 261
10.4 Comparison of Estimators 262
10.4.1 Comparison of RE with Restricted RE 262
10.4.2 Comparison of RE with PTRE 263
10.4.3 Comparison of RE with SRE and PRSRE 263
10.4.4 Comparison of RE and Restricted RE with RRRE 265
10.4.5 Comparison of RRRE with PTRE, SRE, and PRSRE 266
10.4.6 Comparison of RLASSO with RE and Restricted RE 267
10.4.7 Comparison of RLASSO with PTRE, SRE, and PRSRE 267
10.4.8 Comparison of Modified RLASSO with RRRE 268
10.5 Summary and Concluding Remarks 268
11 High-Dimensional Ridge Regression 285
11.1 High-Dimensional RRE 286
11.2 High-Dimensional Stein-Type RRE 288
11.2.1 Numerical Results 291
11.2.1.1 Example: Riboflavin Data 291
11.2.1.2 Monte Carlo Simulation 291
11.3 Post Selection Shrinkage 293
11.3.1 Notation and Assumptions 296
11.3.2 Estimation Strategy 297
11.3.3 Asymptotic Distributional L2-Risks 299
11.4 Summary and Concluding Remarks 300
12 Applications: Neural Networks and Big Data 303
12.1 Introduction 304
12.2 A Simple Two-Layer Neural Network 307
12.2.1 Logistic Regression Revisited 307
12.2.2 Logistic Regression Loss Function with Penalty 310
12.2.3 Two-Layer Logistic Regression 311
12.3 Deep Neural Networks 313
12.4 Application: Image Recognition 315
12.4.1 Background 315
12.4.2 Binary Classification 316
12.4.3 Image Preparation 318
12.4.4 Experimental Results 320
12.5 Summary and Concluding Remarks 323
References 325
Index 333
1
Introduction to Ridge Regression
This chapter reviews the developments of ridge regression, starting with the definition of ridge regression together with the covariance matrix. We discuss the multicollinearity problem and ridge notion and present the preliminary test and Stein-type estimators. In addition, we discuss the high-dimensional problem. In conclusion, we include detailed notes, references, and organization of the book.
1.1 Introduction
Consider the common multiple linear regression model with the vector of coefficients, given by
(1.1)where is a vector of responses, is an design matrix of rank , is the vector of covariates, and is an -vector of independently and identically distributed () random variables ().
The least squares estimator () of , denoted by , can be obtained by minimizing the residual sum of squares (), the convex optimization problem,
where is the RSS. Solving
with respect to () gives
(1.2)Suppose that and for some . Then, the variance-covariance matrix of LSE is given by
(1.3)Now, we consider the canonical form of the multiple linear regression model to illustrate how large eigenvalues of the design matrix may affect the efficiency of estimation.
Write the spectral decomposition of the positive definite design matrix to get , where is a column orthogonal matrix of eigenvectors and , where , is the ordered eigenvalue matrix corresponding to . Then,
(1.4)The LSE of has the form,
(1.5)The variance-covariance matrix of is given by
(1.6)Summation of the diagonal elements of the variance-covariance matrix of is equal to . Apparently, small eigenvalues inflate the total variance of estimate or energy of . Specifically, since the eigenvalues are ordered, if the first eigenvalue is small, it causes the variance to explode. If this happens, what must one do? In the following section, we consider this problem. Therefore, it is of interest to realize when the eigenvalues become small.
Before discussing this problem, a very primitive understanding is that if we enlarge the eigenvalues from to , for some positive value, say, , then we can prevent the total variance from exploding. Of course, the amount of recovery depends on the correct choice of the parameter, .
An artificial remedy is to have based on the variance matrix given by
(1.7)Replacing the eigenvector matrix in (1.5) by this matrix (1.7), we get the and the variance as in (1.8).
(1.8)which shows
(1.9)Further, we show that achieving the total variance of is the target.
1.1.1 Multicollinearity Problem
Multicollinearity or collinearity is the existence of near-linear relationships among the regressors, predictors, or input/exogenous variables. There are terms such as exact, complete and severe, or supercollinearity and moderate collinearity. Supercollinearity indicates that two (or multiple) covariates are linearly dependent, and moderate occurs when covariates are moderately correlated. In the complete collinearity case, the design matrix is not invertible. This case mostly occurs in a high-dimensional situation (e.g. microarray measure) in which the number of covariates () exceeds the number of samples ().
Moderation occurs when the relationship between two variables depends on a third variable, namely, the moderator. This case mostly happens in structural equation modeling. Although moderate multicollinearity does not cause the mathematical problems of complete multicollinearity, it does affect the interpretation of model parameter estimates. According to Montgomery et al. (2012), if there is no linear relationship between the regressors, they are said to be orthogonal.
Multicollinearity or ill-conditioning can create inaccurate estimates of the regression coefficients, inflate the standard errors of the regression coefficients, deflate the partial -tests for the regression coefficients, give false and nonsignificant -values, and degrade the predictability of the model. It also causes changes in the direction of signs of the coefficient estimates. According to Montgomery et al. (2012), there are five sources for multicollinearity: (i) data collection, (ii) physical constraints, (iii) overdefined model, (iv) model choice or specification, and (v) outliers.
There are many studies that well explain the problem of multicollinearity. Since theoretical aspects of ridge regression and related issues are our goal, we refer the reader to Montgomery et al. (2012) for illustrative examples and comprehensive study on the multicollinearity and diagnostic measures such as correlation matrix, eigen system analysis of , known as condition number, or variance decomposition proportion and variance inflation factor (). To end this section, we consider a frequently used example in a ridge regression, namely, the Portland cement data introduced by Woods et al. (1932) from Najarian et al. (2013). This data set has been analyzed by many authors, e.g. Kaciranlar et al. (1999), Kibria (2003), and Arashi et al. (2015). We assemble the data as follows:
(1.10)The following (see Tables 1.1 and 1.2) is the partial output of linear regression fit to this data using the software SPSS, where we selected Enter as the method.
We display the VIF values to diagnose the multicollinearity problem in this data set. Values greater than 10 are a sign of multicollinearity. Many softwares can be used.
Table 1.1 Model fit indices for Portland cement data.
Model Adjacent Standard error of the estimate 1 0.991 0.982 0.974 2.446Table 1.2 Coefficient estimates for Portland cement data.
Coefficient Standardized coefficient Significance VIF (Constant) 0.891 0.399 0.607 2.083 0.071 38.496 0.528 0.705 0.501 254.423 0.043 0.135 0.896 46.868 0.160 0.203 0.844 282.5131.2 Ridge Regression Estimator: Ridge Notion
If the regression coefficients s are unconstrained, then they can explode (become large); this results in high variance. Hence, in order to control the variance, one may regularize the regression coefficients and determine how large the coefficient grows. In other words, one may impose a constraint on them so as not to get unboundedly large or penalized large regression coefficients. One type of constraint is the ridge constraint given by for some positive value . Hence, the minimization of the penalized residual sum of squares () is equivalent to solving the following convex optimization problem,
(1.11)for some positive value .
In general, the PRSS is defined by
(1.12)Since the PRSS is a convex function w.r.t. , it has a unique solution. Because of the ridge constraint, the solution is termed as the ridge regression estimator ().
To derive the RRE, we solve the following convex optimization problem
(1.13)where is the PRSS. Solving
w.r.t. gives the RRE,
(1.14)Here, is the shrinkage (tuning) parameter. Indeed, tunes (controls) the size of the coefficients, and hence regularizes them. As , the RRE simplifies to the LSE. Also, as , the RREs approach zero. Hence, the optimal shrinkage parameter is of interest.
One must note that solving the optimization problem (1.13) is not the only way of yielding the RRE. It can also be obtained by solving a RSS of another data, say augmented data. To be specific, consider the following augmentation approach. Let
(1.15)Assume the following multiple linear model,
(1.16)where is an -vector of i.i.d. random variables. Then, the LSE of is obtained as
(1.17)Thus, the LSE of the augmented data is indeed the RRE of the normal data.
1.3 LSE vs. RRE
Indeed, the RRE is proportional to the LSE. Under the orthonormal case, i.e. , the RRE simplifies to
(1.18)Theorem 1.1
The ridge penalty shrinks the eigenvalues of the design matrix.
Proof
Write the singular value...
