Matrix Differential Calculus with Applications in Statistics and Econometrics
Description
This exhaustive, self-contained book on matrix theory and matrix differential calculus provides a treatment of matrix calculus based on differentials and shows how easy it is to use this theory once you have mastered the technique. Jan Magnus, who, along with the late Heinz Neudecker, pioneered the theory, develops it further in this new edition and provides many examples along the way to support it.
Matrix calculus has become an essential tool for quantitative methods in a large number of applications, ranging from social and behavioral sciences to econometrics. It is still relevant and used today in a wide range of subjects such as the biosciences and psychology. Matrix Differential Calculus with Applications in Statistics and Econometrics, Third Edition contains all of the essentials of multivariable calculus with an emphasis on the use of differentials. It starts by presenting a concise, yet thorough overview of matrix algebra, then goes on to develop the theory of differentials. The rest of the text combines the theory and application of matrix differential calculus, providing the practitioner and researcher with both a quick review and a detailed reference.
* Fulfills the need for an updated and unified treatment of matrix differential calculus
* Contains many new examples and exercises based on questions asked of the author over the years
* Covers new developments in field and features new applications
* Written by a leading expert and pioneer of the theory
* Part of the Wiley Series in Probability and Statistics
Matrix Differential Calculus With Applications in Statistics and Econometrics Third Edition is an ideal text for graduate students and academics studying the subject, as well as for postgraduates and specialists working in biosciences and psychology.
More details
Other editions
Additional editions
Persons
JAN R. MAGNUS is Emeritus Professor at the Department of Econometrics & Operations Research, Tilburg University, and Extraordinary Professor at the Department of Econometrics & Operations Research, Vrije University, Amsterdam. He is research fellow of CentER and the Tinbergen Institute. He has co-authored nine books and is the author of over 100 scientific papers.
HEINZ NEUDECKER (1933-2017) was Professor of Econometrics at the University of Amsterdam from 1972 until his retirement in 1998.
Content
Preface xiii
Part One - Matrices
1 Basic properties of vectors and matrices 3
1 Introduction 3
2 Sets 3
3 Matrices: addition and multiplication 4
4 The transpose of a matrix 6
5 Square matrices 6
6 Linear forms and quadratic forms 7
7 The rank of a matrix 9
8 The inverse 10
9 The determinant 10
10 The trace 11
11 Partitioned matrices 12
12 Complex matrices 14
13 Eigenvalues and eigenvectors 14
14 Schur's decomposition theorem 17
15 The Jordan decomposition 18
16 The singular-value decomposition 20
17 Further results concerning eigenvalues 20
18 Positive (semi)definite matrices 23
19 Three further results for positive definite matrices 25
20 A useful result 26
21 Symmetric matrix functions 27
Miscellaneous exercises 28
Bibliographical notes 30
2 Kronecker products, vec operator, and Moore-Penrose inverse 31
1 Introduction 31
2 The Kronecker product 31
3 Eigenvalues of a Kronecker product 33
4 The vec operator 34
5 The Moore-Penrose (MP) inverse 36
6 Existence and uniqueness of the MP inverse 37
7 Some properties of the MP inverse 38
8 Further properties 39
9 The solution of linear equation systems 41
Miscellaneous exercises 43
Bibliographical notes 45
3 Miscellaneous matrix results 47
1 Introduction 47
2 The adjoint matrix 47
3 Proof of Theorem 3.1 49
4 Bordered determinants 51
5 The matrix equation AX = 0 51
6 The Hadamard product 52
7 The commutation matrix Kmn 54
8 The duplication matrix Dn 56
9 Relationship between Dn+1 and Dn, I 58
10 Relationship between Dn+1 and Dn, II 59
11 Conditions for a quadratic form to be positive (negative) subject to linear constraints 60
12 Necessary and sufficient conditions for r(A : B) = r(A) + r(B) 63
13 The bordered Gramian matrix 65
14 The equations X1A + X2B' = G1,X1B = G2 67
Miscellaneous exercises 69
Bibliographical notes 70
Part Two - Differentials: the theory
4 Mathematical preliminaries 73
1 Introduction 73
2 Interior points and accumulation points 73
3 Open and closed sets 75
4 The Bolzano-Weierstrass theorem 77
5 Functions 78
6 The limit of a function 79
7 Continuous functions and compactness 80
8 Convex sets 81
9 Convex and concave functions 83
Bibliographical notes 86
5 Differentials and differentiability 87
1 Introduction 87
2 Continuity 88
3 Differentiability and linear approximation 90
4 The differential of a vector function 91
5 Uniqueness of the differential 93
6 Continuity of differentiable functions 94
7 Partial derivatives 95
8 The first identification theorem 96
9 Existence of the differential, I 97
10 Existence of the differential, II 99
11 Continuous differentiability 100
12 The chain rule 100
13 Cauchy invariance 102
14 The mean-value theorem for real-valued functions 103
15 Differentiable matrix functions 104
16 Some remarks on notation 106
17 Complex differentiation 108
Miscellaneous exercises 110
Bibliographical notes 110
6 The second differential 111
1 Introduction 111
2 Second-order partial derivatives 111
3 The Hessian matrix 112
4 Twice differentiability and second-order approximation, I 113
5 Definition of twice differentiability 114
6 The second differential 115
7 Symmetry of the Hessian matrix 117
8 The second identification theorem 119
9 Twice differentiability and second-order approximation, II 119
10 Chain rule for Hessian matrices 121
11 The analog for second differentials 123
12 Taylor's theorem for real-valued functions 124
13 Higher-order differentials 125
14 Real analytic functions 125
15 Twice differentiable matrix functions 126
Bibliographical notes 127
7 Static optimization 129
1 Introduction 129
2 Unconstrained optimization 130
3 The existence of absolute extrema 131
4 Necessary conditions for a local minimum 132
5 Sufficient conditions for a local minimum: first-derivative test 134
6 Sufficient conditions for a local minimum: second-derivative test 136
7 Characterization of differentiable convex functions 138
8 Characterization of twice differentiable convex functions 141
9 Sufficient conditions for an absolute minimum 142
10 Monotonic transformations 143
11 Optimization subject to constraints 144
12 Necessary conditions for a local minimum under constraints 145
13 Sufficient conditions for a local minimum under constraints 149
14 Sufficient conditions for an absolute minimum under constraints 154
15 A note on constraints in matrix form 155
16 Economic interpretation of Lagrange multipliers 155
Appendix: the implicit function theorem 157
Bibliographical notes 159
Part Three - Differentials: the practice
8 Some important differentials 163
1 Introduction 163
2 Fundamental rules of differential calculus 163
3 The differential of a determinant 165
4 The differential of an inverse 168
5 Differential of the Moore-Penrose inverse 169
6 The differential of the adjoint matrix 172
7 On differentiating eigenvalues and eigenvectors 174
8 The continuity of eigenprojections 176
9 The differential of eigenvalues and eigenvectors: symmetric case 180
10 Two alternative expressions for d¿ 183
11 Second differential of the eigenvalue function 185
Miscellaneous exercises 186
Bibliographical notes 189
9 First-order differentials and Jacobian matrices 191
1 Introduction 191
2 Classification 192
3 Derisatives 192
4 Derivatives 194
5 Identification of Jacobian matrices 196
6 The first identification table 197
7 Partitioning of the derivative 197
8 Scalar functions of a scalar 198
9 Scalar functions of a vector 198
10 Scalar functions of a matrix, I: trace 199
11 Scalar functions of a matrix, II: determinant 201
12 Scalar functions of a matrix, III: eigenvalue 202
13 Two examples of vector functions 203
14 Matrix functions 204
15 Kronecker products 206
16 Some other problems 208
17 Jacobians of transformations 209
Bibliographical notes 210
10 Second-order differentials and Hessian matrices 211
1 Introduction 211
2 The second identification table 211
3 Linear and quadratic forms 212
4 A useful theorem 213
5 The determinant function 214
6 The eigenvalue function 215
7 Other examples 215
8 Composite functions 217
9 The eigenvector function 218
10 Hessian of matrix functions, I 219
11 Hessian of matrix functions, II 219
Miscellaneous exercises 220
Part Four - Inequalities
11 Inequalities 225
1 Introduction 225
2 The Cauchy-Schwarz inequality 226
3 Matrix analogs of the Cauchy-Schwarz inequality 227
4 The theorem of the arithmetic and geometric means 228
5 The Rayleigh quotient 230
6 Concavity of ¿1 and convexity of ¿n 232
7 Variational description of eigenvalues 232
8 Fischer's min-max theorem 234
9 Monotonicity of the eigenvalues 236
10 The Poincar¿e separation theorem 236
11 Two corollaries of Poincar¿e's theorem 237
12 Further consequences of the Poincar¿e theorem 238
13 Multiplicative version 239
14 The maximum of a bilinear form 241
15 Hadamard's inequality 242
16 An interlude: Karamata's inequality 242
17 Karamata's inequality and eigenvalues 244
18 An inequality concerning positive semidefinite matrices 245
19 A representation theorem for ( ¿api )1/p 246
20 A representation theorem for (trAp)1/p 247
21 Hölder's inequality 248
22 Concavity of log|A| 250
23 Minkowski's inequality 251
24 Quasilinear representation of |A|1/n 253
25 Minkowski's determinant theorem 255
26 Weighted means of order p 256
27 Schlömilch's inequality 258
28 Curvature properties of Mp(x, a) 259
29 Least squares 260
30 Generalized least squares 261
31 Restricted least squares 262
32 Restricted least squares: matrix version 264
Miscellaneous exercises 265
Bibliographical notes 269
Part Five - The linear model
12 Statistical preliminaries 273
1 Introduction 273
2 The cumulative distribution function 273
3 The joint density function 274
4 Expectations 274
5 Variance and covariance 275
6 Independence of two random variables 277
7 Independence of n random variables 279
8 Sampling 279
9 The one-dimensional normal distribution 279
10 The multivariate normal distribution 280
11 Estimation 282
Miscellaneous exercises 282
Bibliographical notes 283
13 The linear regression model 285
1 Introduction 285
2 Affine minimum-trace unbiased estimation 286
3 The Gauss-Markov theorem 287
4 The method of least squares 290
5 Aitken's theorem 291
6 Multicollinearity 293
7 Estimable functions 295
8 Linear constraints: the case M(R') ¿M(X') 296
9 Linear constraints: the general case 300
10 Linear constraints: the case M(R') nM(X') = {0} 302
11 A singular variance matrix: the case M(X) ¿M(V ) 304
12 A singular variance matrix: the case r(X'V +X) = r(X) 305
13 A singular variance matrix: the general case, I 307
14 Explicit and implicit linear constraints 307
15 The general linear model, I 310
16 A singular variance matrix: the general case, II 311
17 The general linear model, II 314
18 Generalized least squares 315
19 Restricted least squares 316
Miscellaneous exercises 318
Bibliographical notes 319
14 Further topics in the linear model 321
1 Introduction 321
2 Best quadratic unbiased estimation of s2 322
3 The best quadratic and positive unbiased estimator of s2 322
4 The best quadratic unbiased estimator of s2 324
5 Best quadratic invariant estimation of s2 326
6 The best quadratic and positive invariant estimator of s2 327
7 The best quadratic invariant estimator of s2 329
8 Best quadratic unbiased estimation: multivariate normal case 330
9 Bounds for the bias of the least-squares estimator of s2, I 332
10 Bounds for the bias of the least-squares estimator of s2, II 333
11 The prediction of disturbances 335
12 Best linear unbiased predictors with scalar variance matrix 336
13 Best linear unbiased predictors with fixed variance matrix, I 338
14 Best linear unbiased predictors with fixed variance matrix, II 340
15 Local sensitivity of the posterior mean 341
16 Local sensitivity of the posterior precision 342
Bibliographical notes 344
Part Six - Applications to maximum likelihood estimation
15 Maximum likelihood estimation 347
1 Introduction 347
2 The method of maximum likelihood (ML) 347
3 ML estimation of the multivariate normal distribution 348
4 Symmetry: implicit versus explicit treatment 350
5 The treatment of positive definiteness 351
6 The information matrix 352
7 ML estimation of the multivariate normal distribution: distinct means 354
8 The multivariate linear regression model 354
9 The errors-in-variables model 357
10 The nonlinear regression model with normal errors 359
11 Special case: functional independence of mean and variance parameters 361
12 Generalization of Theorem 15.6 362
Miscellaneous exercises 364
Bibliographical notes 365
16 Simultaneous equations 367
1 Introduction 367
2 The simultaneous equations model 367
3 The identification problem 369
4 Identification with linear constraints on B and G only 371
5 Identification with linear constraints on B, G, and ¿ 371
6 Nonlinear constraints 373
7 FIML: the information matrix (general case) 374
8 FIML: asymptotic variance matrix (special case) 376
9 LIML: first-order conditions 378
10 LIML: information matrix 381
11 LIML: asymptotic variance matrix 383
Bibliographical notes 388
17 Topics in psychometrics 389
1 Introduction 389
2 Population principal components 390
3 Optimality of principal components 391
4 A related result 392
5 Sample principal components 393
6 Optimality of sample principal components 395
7 One-mode component analysis 395
8 One-mode component analysis and sample principal components 398
9 Two-mode component analysis 399
10 Multimode component analysis 400
11 Factor analysis 404
12 A zigzag routine 407
13 A Newton-Raphson routine 408
14 Kaiser's varimax method 412
15 Canonical correlations and variates in the population 414
16 Correspondence analysis 417
17 Linear discriminant analysis 418
Bibliographical notes 419
Part Seven - Summary
18 Matrix calculus: the essentials 423
1 Introduction 423
2 Differentials 424
3 Vector calculus 426
4 Optimization 429
5 Least squares 431
6 Matrix calculus 432
7 Interlude on linear and quadratic forms 434
8 The second differential 434
9 Chain rule for second differentials 436
10 Four examples 438
11 The Kronecker product and vec operator 439
12 Identification 441
13 The commutation matrix 442
14 From second differential to Hessian 443
15 Symmetry and the duplication matrix 444
16 Maximum likelihood 445
Further reading 448
Bibliography 449
Index of symbols 467
Subject index 471
Preface
Preface to the first edition
There has been a long-felt need for a book that gives a self-contained and unified treatment of matrix differential calculus, specifically written for econometricians and statisticians. The present book is meant to satisfy this need. It can serve as a textbook for advanced undergraduates and postgraduates in econometrics and as a reference book for practicing econometricians. Mathematical statisticians and psychometricians may also find something to their liking in the book.
When used as a textbook, it can provide a full-semester course. Reasonable proficiency in basic matrix theory is assumed, especially with the use of partitioned matrices. The basics of matrix algebra, as deemed necessary for a proper understanding of the main subject of the book, are summarized in Part One, the first of the book's six parts. The book also contains the essentials of multivariable calculus but geared to and often phrased in terms of differentials.
The sequence in which the chapters are being read is not of great consequence. It is fully conceivable that practitioners start with Part Three (Differentials: the practice) and, dependent on their predilections, carry on to Parts Five or Six, which deal with applications. Those who want a full understanding of the underlying theory should read the whole book, although even then they could go through the necessary matrix algebra only when the specific need arises.
Matrix differential calculus as presented in this book is based on differentials, and this sets the book apart from other books in this area. The approach via differentials is, in our opinion, superior to any other existing approach. Our principal idea is that differentials are more congenial to multivariable functions as they crop up in econometrics, mathematical statistics, or psychometrics than derivatives, although from a theoretical point of view the two concepts are equivalent.
The book falls into six parts. Part One deals with matrix algebra. It lists, and also often proves, items like the Schur, Jordan, and singular-value decompositions; concepts like the Hadamard and Kronecker products; the vec operator; the commutation and duplication matrices; and the Moore-Penrose inverse. Results on bordered matrices (and their determinants) and (linearly restricted) quadratic forms are also presented here.
Part Two, which forms the theoretical heart of the book, is entirely devoted to a thorough treatment of the theory of differentials, and presents the essentials of calculus but geared to and phrased in terms of differentials. First and second differentials are defined, 'identification' rules for Jacobian and Hessian matrices are given, and chain rules derived. A separate chapter on the theory of (constrained) optimization in terms of differentials concludes this part.
Part Three is the practical core of the book. It contains the rules for working with differentials, lists the differentials of important scalar, vector, and matrix functions (inter alia eigenvalues, eigenvectors, and the Moore-Penrose inverse) and supplies 'identification' tables for Jacobian and Hessian matrices.
Part Four, treating inequalities, owes its existence to our feeling that econometricians should be conversant with inequalities, such as the Cauchy-Schwarz and Minkowski inequalities (and extensions thereof), and that they should also master a powerful result like Poincaré's separation theorem. This part is to some extent also the case history of a disappointment. When we started writing this book we had the ambition to derive all inequalities by means of matrix differential calculus. After all, every inequality can be rephrased as the solution of an optimization problem. This proved to be an illusion, due to the fact that the Hessian matrix in most cases is singular at the optimum point.
Part Five is entirely devoted to applications of matrix differential calculus to the linear regression model. There is an exhaustive treatment of estimation problems related to the fixed part of the model under various assumptions concerning ranks and (other) constraints. Moreover, it contains topics relating to the stochastic part of the model, viz. estimation of the error variance and prediction of the error term. There is also a small section on sensitivity analysis. An introductory chapter deals with the necessary statistical preliminaries.
Part Six deals with maximum likelihood estimation, which is of course an ideal source for demonstrating the power of the propagated techniques. In the first of three chapters, several models are analysed, inter alia the multivariate normal distribution, the errors-in-variables model, and the nonlinear regression model. There is a discussion on how to deal with symmetry and positive definiteness, and special attention is given to the information matrix. The second chapter in this part deals with simultaneous equations under normality conditions. It investigates both identification and estimation problems, subject to various (non)linear constraints on the parameters. This part also discusses full-information maximum likelihood (FIML) and limited-information maximum likelihood (LIML), with special attention to the derivation of asymptotic variance matrices. The final chapter addresses itself to various psychometric problems, inter alia principal components, multimode component analysis, factor analysis, and canonical correlation.
All chapters contain many exercises. These are frequently meant to be complementary to the main text.
A large number of books and papers have been published on the theory and applications of matrix differential calculus. Without attempting to describe their relative virtues and particularities, the interested reader may wish to consult Dwyer and Macphail (1948), Bodewig (1959), Wilkinson (1965), Dwyer (1967), Neudecker (1967, 1969), Tracy and Dwyer (1969), Tracy and Singh (1972), McDonald and Swaminathan (1973), MacRae (1974), Balestra (1976), Bentler and Lee (1978), Henderson and Searle (1979), Wong and Wong (1979, 1980), Nel (1980), Rogers (1980), Wong (1980, 1985), Graham (1981), McCulloch (1982), Schönemann (1985), Magnus and Neudecker (1985), Pollock (1985), Don (1986), and Kollo (1991). The papers by Henderson and Searle (1979) and Nel (1980), and Rogers' (1980) book contain extensive bibliographies.
The two authors share the responsibility for Parts One, Three, Five, and Six, although any new results in Part One are due to Magnus. Parts Two and Four are due to Magnus, although Neudecker contributed some results to Part Four. Magnus is also responsible for the writing and organization of the final text.
We wish to thank our colleagues F. J. H. Don, R. D. H. Heijmans, D. S. G. Pollock, and R. Ramer for their critical remarks and contributions. The greatest obligation is owed to Sue Kirkbride at the London School of Economics who patiently and cheerfully typed and retyped the various versions of the book. Partial financial support was provided by the Netherlands Organization for the Advancement of Pure Research (Z. W. O.) and the Suntory Toyota International Centre for Economics and Related Disciplines at the London School of Economics.
London/Amsterdam Jan R. Magnus
April 1987 Heinz Neudecker
Preface to the first revised printing
Since this book first appeared - now almost four years ago - many of our colleagues, students, and other readers have pointed out typographical errors and have made suggestions for improving the text. We are particularly grateful to R. D. H. Heijmans, J. F. Kiviet, I. J. Steyn, and G. Trenkler. We owe the greatest debt to F. Gerrish, formerly of the School of Mathematics in the Polytechnic, Kingston-upon-Thames, who read Chapters 1-11 with awesome precision and care and made numerous insightful suggestions and constructive remarks. We hope that this printing will continue to trigger comments from our readers.
London/Tilburg/Amsterdam Jan R. Magnus
February 1991 Heinz Neudecker
Preface to the second edition
A further seven years have passed since our first revision in 1991. We are happy to see that our book is still being used by colleagues and students. In this revision we attempted to reach three goals. First, we made a serious attempt to keep the book up-to-date by adding many recent references and new exercises. Second, we made numerous small changes throughout the text, improving the clarity of exposition. Finally, we corrected a number of typographical and other errors.
The structure of the book and its philosophy are unchanged. Apart from a large number of small changes, there are two major changes. First, we interchanged Sections 12 and 13 of Chapter 1, since complex numbers need to be discussed before eigenvalues and eigenvectors, and we corrected an error in Theorem 1.7. Second, in Chapter 17 on psychometrics, we rewrote Sections 8-10 relating to the Eckart-Young theorem.
We are grateful to Karim Abadir, Paul Bekker, Hamparsum Bozdogan, Michael Browne, Frank Gerrish, Kaddour Hadri, Tõnu Kollo, Shuangzhe Liu, Daan Nel, Albert Satorra, Kazuo Shigemasu, Jos ten Berge, Peter ter Berg, Götz Trenkler, Haruo Yanai, and many others for their thoughtful and constructive comments. Of course, we welcome further comments from our readers.
Tilburg/Amsterdam Jan R. Magnus
March 1998 Heinz Neudecker
Preface to the third edition
Twenty years have passed since the appearance of...
