Matrix Algebra for Linear Models
Description
Reviews / Votes
"This book seems suitable for an advanced undergraduate and/or introductory master's level course . . . Four appealing features of this book are its inclusion of an overview, a summary, exercises (with answers provided), and numerical examples for all sections." (American Mathematical Society, 1 November 2015) "The book is suitable for graduate and postgraduate students and researchers. This book is highly recommended." (Zentralblatt, 1 April 2015) "This is an excellent and comprehensive presentation of the use of matrices for linear models. The writing is very clear, and the layout is excellent. It would serve well either as a class text or as the foundation for individual personal study." (International Statistical Review, 18 March 2014)More details
Other editions
Additional editions
Person
Content
Preface xiii
Acknowledgments xv
Part I Basic Ideas about Matrices and Systems of Linear Equations 1
Section 1 What Matrices are and Some Basic Operations with Them 3
1.1 Introduction 3
1.2 What are Matrices and why are they Interesting to a Statistician? 3
1.3 Matrix Notation Addition and Multiplication 6
1.4 Summary 10
Exercises 10
Section 2 Determinants and Solving a System of Equations 14
2.1 Introduction 14
2.2 Definition of and Formulae for Expanding Determinants 14
2.3 Some Computational Tricks for the Evaluation of Determinants 16
2.4 Solution to Linear Equations Using Determinants 18
2.5 Gauss Elimination 22
2.6 Summary 27
Exercises 27
Section 3 The Inverse of a Matrix 30
3.1 Introduction 30
3.2 The Adjoint Method of Finding the Inverse of a Matrix 30
3.3 Using Elementary Row Operations 31
3.4 Using the Matrix Inverse to Solve a System of Equations 33
3.5 Partitioned Matrices and Their Inverses 34
3.6 Finding the Least Square Estimator 38
3.7 Summary 44
Exercises 44
Section 4 Special Matrices and Facts about Matrices that will be used in the Sequel 47
4.1 Introduction 47
4.2 Matrices of the Form aIn + bJn 47
4.3 Orthogonal Matrices 49
4.4 Direct Product of Matrices 52
4.5 An Important Property of Determinants 53
4.6 The Trace of a Matrix 56
4.7 Matrix Differentiation 57
4.8 The Least Square Estimator Again 62
4.9 Summary 62
Exercises 63
Section 5 Vector Spaces 66
5.1 Introduction 66
5.2 What is a Vector Space? 66
5.3 The Dimension of a Vector Space 68
5.4 Inner Product Spaces 70
5.5 Linear Transformations 73
5.6 Summary 76
Exercises 76
Section 6 The Rank of a Matrix and Solutions to Systems of Equations 79
6.1 Introduction 79
6.2 The Rank of a Matrix 79
6.3 Solving Systems of Equations with Coefficient Matrix of Less than Full Rank 84
6.4 Summary 87
Exercises 87
Part II Eigenvalues the Singular Value Decomposition and Principal Components 91
Section 7 Finding the Eigenvalues of a Matrix 93
7.1 Introduction 93
7.2 Eigenvalues and Eigenvectors of a Matrix 93
7.3 Nonnegative Definite Matrices 101
7.4 Summary 104
Exercises 105
Section 8 The Eigenvalues and Eigenvectors of Special Matrices 108
8.1 Introduction 108
8.2 Orthogonal Nonsingular and Idempotent Matrices 109
8.3 The Cayley-Hamilton Theorem 112
8.4 The Relationship between the Trace the Determinant and the Eigenvalues of a Matrix 114
8.5 The Eigenvalues and Eigenvectors of the Kronecker Product of Two Matrices 116
8.6 The Eigenvalues and the Eigenvectors of a Matrix of the Form aI + bJ 117
8.7 The Loewner Ordering 119
8.8 Summary 121
Exercises 122
Section 9 The Singular Value Decomposition (SVD) 124
9.1 Introduction 124
9.2 The Existence of the SVD 125
9.3 Uses and Examples of the SVD 127
9.4 Summary 134
Exercises 134
Section 10 Applications of the Singular Value Decomposition 137
10.1 Introduction 137
10.2 Reparameterization of a Non-full-Rank Model to a Full-Rank Model 137
10.3 Principal Components 141
10.4 The Multicollinearity Problem 143
10.5 Summary 144
Exercises 145
Section 11 Relative Eigenvalues and Generalizations of the Singular Value Decomposition 146
11.1 Introduction 146
11.2 Relative Eigenvalues and Eigenvectors 146
11.3 Generalizations of the Singular Value Decomposition:Overview 151
11.4 The First Generalization 152
11.5 The Second Generalization 157
11.6 Summary 160
Exercises 160
Part III Generalized Inverses 163
Section 12 Basic Ideas about Generalized Inverses 165
12.1 Introduction 165
12.2 What is a Generalized Inverse and how is One Obtained? 165
12.3 The Moore-Penrose Inverse 170
12.4 Summary 173
Exercises 173
Section 13 Characterizations of Generalized Inverses Using the Singular Value Decomposition 175
13.1 Introduction 175
13.2 Characterization of the Moore-Penrose Inverse 175
13.3 Generalized Inverses in Terms of the Moore-Penrose Inverse 177
13.4 Summary 185
Exercises 186
Section 14 Least Square and Minimum Norm Generalized Inverses 188
14.1 Introduction 188
14.2 Minimum Norm Generalized Inverses 189
14.3 Least Square Generalized Inverses 193
14.4 An Extension of Theorem 7.3 to Positive-Semi-definite Matrices 196
14.5 Summary 197
Exercises 197
Section 15 More Representations of Generalized Inverses 200
15.1 Introduction 200
15.2 Another Characterization of the Moore-Penrose Inverse 200
15.3 Still another Representation of the Generalized Inverse 204
15.4 The Generalized Inverse of a Partitioned Matrix 207
15.5 Summary 211
Exercises 211
Section 16 Least Square Estimators for Less than Full-Rank Models 213
16.1 Introduction 213
16.2 Some Preliminaries 213
16.3 Obtaining the LS Estimator 214
16.4 Summary 221
Exercises 221
Part IV Quadratic Forms and the Analysis of Variance 223
Section 17 Quadratic Forms and their Probability Distributions 225
17.1 Introduction 225
17.2 Examples of Quadratic Forms 225
17.3 The Chi-Square Distribution 228
17.4 When does the Quadratic Form of a Random Variable have a Chi-Square Distribution? 230
17.5 When are Two Quadratic Forms with the Chi-Square Distribution Independent? 231
17.6 Summary 234
Exercises 235
Section 18 Analysis of Variance: Regression Models and the One- and Two-Way Classification 237
18.1 Introduction 237
18.2 The Full-Rank General Linear Regression Model 237
18.3 Analysis of Variance: One-Way Classification 241
18.4 Analysis of Variance: Two-Way Classification 244
18.5 Summary 249
Exercises 249
Section 19 More ANOVA 253
19.1 Introduction 253
19.2 The Two-Way Classification with Interaction 254
19.3 The Two-Way Classification with One Factor Nested 258
19.4 Summary 262
Exercises 262
Section 20 The General Linear Hypothesis 264
20.1 Introduction 264
20.2 The Full-Rank Case 264
20.3 The Non-full-Rank Case 267
20.4 Contrasts 270
20.5 Summary 273
Exercises 273
Part V Matrix Optimization Problems 275
Section 21 Unconstrained Optimization Problems 277
21.1 Introduction 277
21.2 Unconstrained Optimization Problems 277
21.3 The Least Square Estimator Again 281
21.4 Summary 283
Exercises 283
Section 22 Constrained Minimization Problems with Linear Constraints 287
22.1 Introduction 287
22.2 An Overview of Lagrange Multipliers 287
22.3 Minimizing a Second-Degree Form with Respect to a Linear Constraint 293
22.4 The Constrained Least Square Estimator 295
22.5 Canonical Correlation 299
22.6 Summary 302
Exercises 302
Section 23 The Gauss-Markov Theorem 304
23.1 Introduction 304
23.2 The Gauss-Markov Theorem and the Least Square Estimator 304
23.3 The Modified Gauss-Markov Theorem and the Linear Bayes Estimator 306
23.4 Summary 311
Exercises 311
Section 24 Ridge Regression-Type Estimators 314
24.1 Introduction 314
24.2 Minimizing a Second-Degree Form with Respect to a Quadratic Constraint 314
24.3 The Generalized Ridge Regression Estimators 315
24.4 The Mean Square Error of the Generalized Ridge Estimator without Averaging over the Prior Distribution 317
24.5 The Mean Square Error Averaging over the Prior Distribution 321
24.6 Summary 321
Exercises 321
Answers to Selected Exercises 324
References 366
Index 368
SECTION 1
WHAT MATRICES ARE AND SOME BASIC OPERATIONS WITH THEM
1.1 INTRODUCTION
This section will introduce matrices and show how they are useful to represent data. It will review some basic matrix operations including matrix addition and multiplication. Some examples to illustrate why they are interesting and important for statistical applications will be given. The representation of a linear model using matrices will be shown.
1.2 WHAT ARE MATRICES AND WHY ARE THEY INTERESTING TO A STATISTICIAN?
Matrices are rectangular arrays of numbers. Some examples of such arrays are
Often data may be represented conveniently by a matrix. We give an example to illustrate how.
Example 1.1 Representing Data by Matrices
An example that lends itself to statistical analysis is taken from the Economic Report of the President of the United States in 1988. The data represent the relationship between a dependent variable Y (personal consumption expenditures) and three other independent variables X1, X2, and X3. The variable X1 represents the gross national product, X2 represents personal income (in billions of dollars), and X3 represents the total number of employed people in the civilian labor force (in thousands). Consider this data for the years 1970–1974 in Table 1.1.
TABLE 1.1 Consumption expenditures in terms of gross national product, personal income, and total number of employed people
The dependent variable may be represented by a matrix with five rows and one column. The independent variables could be represented by a matrix with five rows and three columns. Thus,
A matrix with m rows and n columns is an m × n matrix. Thus, the matrix Y in Example 1.1 is 5 × 1 and the matrix X is 5 × 3. A square matrix is one that has the same number of rows and columns. The individual numbers in a matrix are called the elements of the matrix.
We now give an example of an application from probability theory that uses matrices.
Example 1.2 A “Musical Room” Problem
Another somewhat different example is the following. Consider a triangular-shaped building with four rooms one at the center, room 0, and three rooms around it numbered 1, 2, and 3 clockwise (Fig. 1.1).
There is a door from room 0 to rooms 1, 2, and 3 and doors connecting rooms 1 and 2, 2 and 3, and 3 and 1. There is a person in the building. The room that he/she is in is the state of the system. At fixed intervals of time, he/she rolls a die. If he/she is in room 0 and the outcome is 1 or 2, he/she goes to room 1. If the outcome is 3 or 4, he/she goes to room 2. If the outcome is 5 or 6, he/she goes to room 3. If the person is in room 1, 2, or 3 and the outcome is 1 or 2, he/she advances one room in the clockwise direction. If the outcome is 3 or 4, he/she advances one room in the counterclockwise direction. An outcome of 5 or 6 will cause the person to return to room 0. Assume the die is fair.
FIGURE 1.1 Building with four rooms.
Let pij be the probability that the person goes from room i to room j. Then we have the table of transitions
that indicates
Then the transition matrix would be
Matrices turn out to be handy for representing data. Equations involving matrices are often used to study the relationship between variables.
More explanation of how this is done will be offered in the sections of the book that follow.
The matrices to be studied in this book will have elements that are real numbers. This will suffice for the study of linear models and many other topics in statistics. We will not consider matrices whose elements are complex numbers or elements of an arbitrary ring or field.
We now consider some basic operations using matrices.
1.3 MATRIX NOTATION, ADDITION, AND MULTIPLICATION
We will show how to represent a matrix and how to add and multiply two matrices.
The elements of a matrix A are denoted by aij meaning the element in the ith row and the jth column. For example, for the matrix
c11 = 0.2, c12 = 0.5, and so on. Three important operations include matrix addition, multiplication of a matrix by a scalar, and matrix multiplication. Two matrices A and B can be added only when they have the same number of rows and columns. For the matrix C = A + B, cij = aij + bij; in other words, just add the elements algebraically in the same row and column. The matrix D = αA where α is a real number has elements dij = αaij; just multiply each element by the scalar. Two matrices can be multiplied only when the number of columns of the first matrix is the same as the number of rows of the second one in the product. The elements of the n × p matrix E = AB, assuming that A is n × m and B is m × p, are
Example 1.3 Illustration of Matrix Operations
Let .
Then
and
Example 1.4 Continuation of Example 1.2
Suppose that elements of the row vector where represent the probability that the person starts in room i. Then π(1) = π(0)P. For example, if
the probabilities the person is in room 0 initially are 1/2, room 1 1/6, room 2 1/12, and room 3 1/4, then
Thus, after one transition given the initial probability vector above the probabilities that the person ends up in room 0, room 1, room 2, or room 3 after one transition are 1/6, 5/18, 11/36, and 1/4, respectively. This example illustrates a discrete Markov chain. The possible transitions are represented as elements of a matrix.
Suppose we want to know the probabilities that a person goes from room i to room j after two transitions. Assuming that what happens at each transition is independent, we could multiply the two matrices. Then
Thus, for example, if the person is in room 1, the probability that he/she returns there after two transitions is 1/3. The probability that he/she winds up in room 3 is 2/9. Also when π(0) is the initial probability vector, we have that π(2) = π(1)P = π(0)P2. The reader is asked to find π(2) in Exercise 1.17.
Two matrices are equal if and only if their corresponding elements are equal. More formally, A = B if and only if aij = bij for all 1 ≤ i ≤ m and 1 ≤ j ≤ n.
Most, but not all, of the rules for addition and multiplication of real numbers hold true for matrices. The associative and commutative laws hold true for addition. The zero matrix is the matrix with all of the elements zero. An additive inverse of a matrix A would be −A, the matrix whose elements are (−1)aij. The distributive laws hold true.
However, there are several properties of real numbers that do not hold true for matrices. First, it is possible to have divisors of zero. It is not hard to find matrices A and B where AB = 0 and neither A or B is the zero matrix (see Example 1.4).
In addition the cancellation rule does not hold true. For real nonzero numbers a, b, c, ba = ca would imply that b = c. However (see Example 1.5) for matrices, BA = CA may not imply that B = C.
Not every matrix has a multiplicative inverse. The identity matrix denoted by I has all ones on the longest (main) diagonal (aij = 1) and zeros elsewhere (aij = 0, i ≠ j). For a matrix A, a multiplicative inverse would be a matrix such that AB = I and BA = I. Furthermore, for matrices A and B, it is not often true that AB = BA. In other words, matrices do not satisfy the commutative law of multiplication in general.
The transpose of a matrix A is the matrix A′ where the rows and the columns of A are exchanged. For example, for the matrix A in Example 1.3,
A matrix A is symmetric when A = A′. If A = − A′, the matrix is said to be skew symmetric. Symmetric matrices come up often in statistics.
Example 1.5 Two Nonzero Matrices Whose Product Is Zero
Consider the matrix
Notice that
Example 1.6 The Cancellation Law for Real Numbers Does Not Hold for Matrices
Consider matrices A, B, C where
Now
but B ≠ C.
Matrix theory is basic to the study of linear models. Example 1.7 indicates how the basic matrix operations studied so far are used in this context.
Example 1.7 The Linear Model
Let Y be an n-dimensional vector of observations, an n × 1 matrix. Let X be an n × m matrix where each column has the values of a prediction variable. It is assumed here that there are m predictors. Let β be an m × 1 matrix of parameters to be estimated. The prediction of the observations will not be exact. Thus, we also need an n-dimensional column vector of errors ε. The general linear model will take the form
(1.1)
Suppose that there are five observations and three prediction variables. Then n = 5 and m = 3. As a result, we would have the multiple regression equation
(1.2)
Equation (1.2) may be represented by the matrix equation
(1.3)
In experimental design models, the matrix is frequently zeros and ones indicating the level of a factor. An example of such a model would be
(1.4)
This is an unbalanced one-way analysis of variance (ANOVA) model where there are three treatments with four observations of treatment 1, three observations of treatment 2, and two observations of treatment 3....
