Matrix Analysis for Statistics
Description
More details
Other editions
Additional editions
Person
Content
Preface xi
About the Companion Website xv
1 A Review of Elementary Matrix Algebra 1
1.1 Introduction 1
1.2 Definitions and Notation 1
1.3 Matrix Addition and Multiplication 2
1.4 The Transpose 3
1.5 The Trace 4
1.6 The Determinant 5
1.7 The Inverse 9
1.8 Partitioned Matrices 12
1.9 The Rank of a Matrix 14
1.10 Orthogonal Matrices 15
1.11 Quadratic Forms 16
1.12 Complex Matrices 18
1.13 Random Vectors and Some Related Statistical Concepts 19
Problems 29
2 Vector Spaces 35
2.1 Introduction 35
2.2 Definitions 35
2.3 Linear Independence and Dependence 42
2.4 Matrix Rank and Linear Independence 45
2.5 Bases and Dimension 49
2.6 Orthonormal Bases and Projections 53
2.7 Projection Matrices 58
2.8 Linear Transformations and Systems of Linear Equations 65
2.9 The Intersection and Sum of Vector Spaces 73
2.10 Oblique Projections 76
2.11 Convex Sets 80
Problems 85
3 Eigenvalues and Eigenvectors 95
3.1 Introduction 95
3.2 Eigenvalues, Eigenvectors, and Eigenspaces 95
3.3 Some Basic Properties of Eigenvalues and Eigenvectors 99
3.4 Symmetric Matrices 106
3.5 Continuity of Eigenvalues and Eigenprojections 114
3.6 Extremal Properties of Eigenvalues 116
3.7 Additional Results Concerning Eigenvalues Of Symmetric Matrices 123
3.8 Nonnegative Definite Matrices 129
3.9 Antieigenvalues and Antieigenvectors 141
Problems 144
4 Matrix Factorizations and Matrix Norms 155
4.1 Introduction 155
4.2 The Singular Value Decomposition 155
4.3 The Spectral Decomposition of a Symmetric Matrix 162
4.4 The Diagonalization of a Square Matrix 169
4.5 The Jordan Decomposition 173
4.6 The Schur Decomposition 175
4.7 The Simultaneous Diagonalization of Two Symmetric Matrices 178
4.8 Matrix Norms 184
Problems 191
5 Generalized Inverses 201
5.1 Introduction 201
5.2 The Moore-Penrose Generalized Inverse 202
5.3 Some Basic Properties of the Moore-Penrose Inverse 205
5.4 The Moore-Penrose Inverse of a Matrix Product 211
5.5 The Moore-Penrose Inverse of Partitioned Matrices 215
5.6 The Moore-Penrose Inverse of a Sum 219
5.7 The Continuity of the Moore-Penrose Inverse 222
5.8 Some Other Generalized Inverses 224
5.9 Computing Generalized Inverses 232
Problems 238
6 Systems of Linear Equations 247
6.1 Introduction 247
6.2 Consistency of a System of Equations 247
6.3 Solutions to a Consistent System of Equations 251
6.4 Homogeneous Systems of Equations 258
6.5 Least Squares Solutions to a System of Linear Equations 260
6.6 Least Squares Estimation For Less Than Full Rank Models 266
6.7 Systems of Linear Equations and The Singular Value Decomposition 271
6.8 Sparse Linear Systems of Equations 273
Problems 278
7 Partitioned Matrices 285
7.1 Introduction 285
7.2 The Inverse 285
7.3 The Determinant 288
7.4 Rank 296
7.5 Generalized Inverses 298
7.6 Eigenvalues 302
Problems 307
8 Special Matrices and Matrix Operations 315
8.1 Introduction 315
8.2 The Kronecker Product 315
8.3 The Direct Sum 323
8.4 The Vec Operator 323
8.5 The Hadamard Product 329
8.6 The Commutation Matrix 339
8.7 Some Other Matrices Associated With the Vec Operator 346
8.8 Nonnegative Matrices 351
8.9 Circulant and Toeplitz Matrices 363
8.10 Hadamard and Vandermonde Matrices 369
Problems 373
9 Matrix Derivatives and Related Topics 387
9.1 Introduction 387
9.2 Multivariable Differential Calculus 387
9.3 Vector and Matrix Functions 390
9.4 Some Useful Matrix Derivatives 396
9.5 Derivatives of Functions of Patterned Matrices 400
9.6 The Perturbation Method 402
9.7 Maxima and Minima 409
9.8 Convex and Concave Functions 413
9.9 The Method of Lagrange Multipliers 417
Problems 423
10 Inequalities 433
10.1 Introduction 433
10.2 Majorization 433
10.3 Cauchy-Schwarz Inequalities 444
10.4 H¿older's Inequality 446
10.5 Minkowski's Inequality 450
10.6 The Arithmetic-Geometric Mean Inequality 452
Problems 453
11 Some Special Topics Related to Quadratic Forms 457
11.1 Introduction 457
11.2 Some Results on Idempotent Matrices 457
11.3 Cochran's Theorem 462
11.4 Distribution of Quadratic Forms in Normal Variates 465
11.5 Independence of Quadratic Forms 471
11.6 Expected Values of Quadratic Forms 477
11.7 The Wishart Distribution 485
Problems 496
References 507
Index 513
CHAPTER 1
A REVIEW OF ELEMENTARY MATRIX ALGEBRA
1.1 Introduction
In this chapter, we review some of the basic operations and fundamental properties involved in matrix algebra. In most cases, properties will be stated without proof, but in some cases, when instructive, proofs will be presented. We end the chapter with a brief discussion of random variables and random vectors, expected values of random variables, and some important distributions encountered elsewhere in the book.
1.2 Definitions and Notation
Except when stated otherwise, a scalar such as a will represent a real number. A matrix A of size m × n is the m × n rectangular array of scalars given by
and sometimes it is simply identified as . Sometimes it also will be convenient to refer to the th element of A, as ; that is, . If , then A is called a square matrix of order m, whereas A is referred to as a rectangular matrix when . An m × 1 matrix
is called a column vector or simply a vector. The element is referred to as the ith component of a. A matrix is called a row vector. The ith row and jth column of the matrix A will be denoted by and , respectively. We will usually use capital letters to represent matrices and lowercase bold letters for vectors.
The diagonal elements of the m × m matrix A are . If all other elements of A are equal to 0, A is called a diagonal matrix and can be identified as . If, in addition, for so that , then the matrix A is called the identity matrix of order m and will be written as or simply if the order is obvious. If and b is a scalar, then we will use to denote the diagonal matrix . For any m × m matrix A, will denote the diagonal matrix with diagonal elements equal to those of A, and for any m × 1 vector a, denotes the diagonal matrix with diagonal elements equal to the components of a; that is, and .
A triangular matrix is a square matrix that is either an upper triangular matrix or a lower triangular matrix. An upper triangular matrix is one that has all of its elements below the diagonal equal to 0, whereas a lower triangular matrix has all of its elements above the diagonal equal to 0. A strictly upper triangular matrix is an upper triangular matrix that has each of its diagonal elements equal to 0. A strictly lower triangular matrix is defined similarly.
The ith column of the m × m identity matrix will be denoted by ei; that is, ei is the m × 1 vector that has its ith component equal to 1 and all of its other components equal to 0. When the value of m is not obvious, we will make it more explicit by writing ei as . The m × m matrix whose only nonzero element is a 1 in the th position will be identified as .
The scalar zero is written 0, whereas a vector of zeros, called a null vector, will be denoted by 0 , and a matrix of zeros, called a null matrix, will be denoted by . The m × 1 vector having each component equal to 1 will be denoted by or simply 1 when the size of the vector is obvious.
1.3 Matrix Addition and Multiplication
The sum of two matrices A and B is defined if they have the same number of rows and the same number of columns; in this case,
The product of a scalar a and a matrix A is
The premultiplication of the matrix B by the matrix A is defined only if the number of columns of A equals the number of rows of B. Thus, if A is and B is , then will be the m × n matrix which has its th element, , given by
A similar definition exists for BA, the postmultiplication of B by A, if the number of columns of B equals the number of rows of A. When both products are defined, we will not have, in general, . If the matrix A is square, then the product AA, or simply , is defined. In this case, if we have , then A is said to be an idempotent matrix.
The following basic properties of matrix addition and multiplication in Theorem 1.1 are easy to verify.
Theorem 1.1
Let a and ß be scalars and A, B, and C be matrices. Then, when the operations involved are defined, the following properties hold:
- a. .
- b. .
- c. .
- d. .
- e. .
- f. .
- g. .
- h. .
1.4 The Transpose
The transpose of an m × n matrix A is the n × m matrix obtained by interchanging the rows and columns of A. Thus, the th element of is . If A is and B is , then the th element of can be expressed as
Thus, evidently . This property along with some other results involving the transpose are summarized in Theorem 1.2.
Theorem 1.2
Let a and ß be scalars and A and B be matrices. Then, when defined, the following properties hold:
- a. .
- b. .
- c. .
- d. .
If A is m × m, that is, A is a square matrix, then is also m × m. In this case, if , then A is called a symmetric matrix, whereas A is called a skew-symmetric if .
The transpose of a column vector is a row vector, and in some situations, we may write a matrix as a column vector times a row vector. For instance, the matrix defined in Section 1.2 can be expressed as . More generally, yields an m × n matrix having 1, as its only nonzero element, in the th position, and if A is an m × n matrix, then
1.5 The Trace
The trace is a function that is defined only on square matrices. If A is an m × m matrix, then the trace of A, denoted by , is defined to be the sum of the diagonal elements of A; that is,
Now if A is m × n and B is n × m, then AB is m × m and
This property of the trace, along with some others, is summarized in Theorem 1.3.
Theorem 1.3
Let a be a scalar and A and B be matrices. Then, when the appropriate operations are defined, we have the following properties:
- a. .
- b. .
- c. .
- d. .
- e. if and only if .
1.6 The Determinant
The determinant is another function defined on square matrices. If A is an m × m matrix, then its determinant, denoted by , is given by
where the summation is taken over all permutations of the set of integers , and the function equals the number of transpositions necessary to change to an increasing sequence of components, that is, to . A transposition is the interchange of two of the integers. Although f is not unique, it is uniquely even or odd, so that is uniquely defined. Note that the determinant produces all products of m terms of the elements of the matrix A such that exactly one element is selected from each row and each column of A.
Using the formula for the determinant, we find that when . If A is 2×2, we have
and when A is 3×3, we get
The following properties of the determinant in Theorem 1.4 are fairly straightforward to verify using the definition of a determinant.
Theorem 1.4
If a is a scalar and A is an m × m matrix, then the following properties hold:
- a. .
- b. .
- c. If A is a diagonal matrix, then .
- d. If all elements of a row (or column) of A are zero, .
- e. The interchange of two rows (or columns) of A changes the sign of .
- f. If all elements of a row (or column) of A are multiplied by a, then the determinant is multiplied by a.
- g. The determinant of A is unchanged when a multiple of one row (or column) is added to another row (or column).
- h. If two rows (or columns) of A are proportional to one another, .
An alternative expression for can be given in terms of the cofactors of A. The minor of the element , denoted by , is the determinant of the matrix obtained after removing the ith row and jth column from A. The corresponding cofactor of , denoted by , is then given as .
Theorem 1.5
For any , the determinant of the m × m matrix A can be obtained by expanding along the ith row,
1.1or expanding along the ith column,
1.2Proof
We will just prove (1.1), as (1.2) can easily be obtained by applying (1.1) to . We...
