Linear Models
Description
More details
Other editions
Additional editions
Persons
The late SHAYLE R. SEARLE, PhD, was Professor Emeritus of Biometry at Cornell University. He was the author of the first edition of Linear Models, Linear Models for Unbalanced Data, and Generalized, Linear, and Mixed Models (with Charles E. McCulloch), all from Wiley. The first edition of Linear Models appears in the Wiley Classics Library.
MARVIN H. J. GRUBER, PhD, is Professor Emeritus at Rochester Institute of Technology, School of Mathematical Sciences. Dr. Gruber has written a number of papers and has given numerous presentations at professional meetings during his tenure as a professor at RIT. His fields of interest include regression estimators and the improvement of their efficiency using shrinkage estimators. He has written and published two books on this topic. Another of his books, Matrix Algebra for Linear Models, also published by Wiley, provides good preparation for studying Linear Models. He is a member of the American Mathematical Society, the Institute of Mathematical Statistics and the American Statistical Association.
Content
Preface to First Edition xxi
About the Companion Website xxv
Introduction and Overview 1
1. Generalized Inverse Matrices 7
1. Introduction, 7
a. Definition and Existence of a Generalized Inverse, 8
b. An Algorithm for Obtaining a Generalized Inverse, 11
c. Obtaining Generalized Inverses Using the Singular Value Decomposition (SVD), 14
2. Solving Linear Equations, 17
a. Consistent Equations, 17
b. Obtaining Solutions, 18
c. Properties of Solutions, 20
3. The Penrose Inverse, 26
4. Other Definitions, 30
5. Symmetric Matrices, 32
a. Properties of a Generalized Inverse, 32
b. Two More Generalized Inverses of X'X, 35
6. Arbitrariness in a Generalized Inverse, 37
7. Other Results, 42
8. Exercises, 44
2. Distributions and Quadratic Forms 49
1. Introduction, 49
2. Symmetric Matrices, 52
3. Positive Definiteness, 53
4. Distributions, 58
a. Multivariate Density Functions, 58
b. Moments, 59
c. Linear Transformations, 60
d. Moment and Cumulative Generating Functions, 62
e. Univariate Normal, 64
f. Multivariate Normal, 64
g. Central ¿2, F, and t, 69
h. Non-central ¿2, 71
i. Non-central F, 73
j. The Non-central t Distribution, 73
5. Distribution of Quadratic Forms, 74
a. Cumulants, 75
b. Distributions, 78
c. Independence, 80
6. Bilinear Forms, 87
7. Exercises, 89
3. Regression for the Full-Rank Model 95
1. Introduction, 95
a. The Model, 95
b. Observations, 97
c. Estimation, 98
d. The General Case of k x Variables, 100
e. Intercept and No-Intercept Models, 104
2. Deviations From Means, 105
3. Some Methods of Estimation, 109
a. Ordinary Least Squares, 109
b. Generalized Least Squares, 109
c. Maximum Likelihood, 110
d. The Best Linear Unbiased Estimator (b.l.u.e.) (Gauss-Markov Theorem), 110
e. Least-squares Theory When The Parameters are Random Variables, 112
4. Consequences of Estimation, 115
a. Unbiasedness, 115
b. Variances, 115
c. Estimating E(y), 116
d. Residual Error Sum of Squares, 119
e. Estimating the Residual Error Variance, 120
f. Partitioning the Total Sum of Squares, 121
g. Multiple Correlation, 122
5. Distributional Properties, 126
a. The Vector of Observations y is Normal, 126
b. The Least-square Estimator b is Normal, 127
c. The Least-square Estimator b and the Estimator of the Variance ^s2 are Independent, 127
d. The Distribution of SSE/s2 is a ¿2 Distribution, 128
e. Non-central ¿2's, 128
f. F-distributions, 129
g. Analyses of Variance, 129
h. Tests of Hypotheses, 131
i. Confidence Intervals, 133
j. More Examples, 136
k. Pure Error, 139
6. The General Linear Hypothesis, 141
a. Testing Linear Hypothesis, 141
b. Estimation Under the Null Hypothesis, 143
c. Four Common Hypotheses, 145
d. Reduced Models, 148
e. Stochastic Constraints, 158
f. Exact Quadratic Constraints (Ridge Regression), 160
7. Related Topics, 162
a. The Likelihood Ratio Test, 163
b. Type I and Type II Errors, 164
c. The Power of a Test, 165
d. Estimating Residuals, 166
8. Summary of Regression Calculations, 168
9. Exercises, 169
4. Introducing Linear Models: Regression on Dummy Variables 175
1. Regression on Allocated Codes, 175
a. Allocated Codes, 175
b. Difficulties and Criticism, 176
c. Grouped Variables, 177
d. Unbalanced Data, 178
2. Regression on Dummy (0, 1) Variables, 180
a. Factors and Levels, 180
b. The Regression, 181
3. Describing Linear Models, 184
a. A One-Way Classification, 184
b. A Two-Way Classification, 186
c. A Three-Way Classification, 188
d. Main Effects and Interactions, 188
e. Nested and Crossed Classifications, 194
4. The Normal Equations, 198
5. Exercises, 201
5. Models Not of Full Rank 205
1. The Normal Equations, 205
a. The Normal Equations, 206
b. Solutions to the Normal Equations, 209
2. Consequences of a Solution, 210
a. Expected Value of bº, 210
b. Variance Covariance Matrices of bº (Variance Covariance Matrices), 211
c. Estimating E(y), 212
d. Residual Error Sum of Squares, 212
e. Estimating the Residual Error Variance, 213
f. Partitioning the Total Sum of Squares, 214
g. Coefficient of Determination, 215
3. Distributional Properties, 217
a. The Observation Vector y is Normal, 217
b. The Solution to the Normal Equations bº is Normally Distributed, 217
c. The Solution to the Normal Equations bº and the Estimator of the Residual Error Variance ^s2 are Independent, 217
d. The Error Sum of Squares Divided by the Population Variance SSE/s2 is Chi-square ¿2, 217
e. Non-central ¿2's, 218
f. Non-central F-distributions, 219
g. Analyses of Variance, 220
h. Tests of Hypotheses, 221
4. Estimable Functions, 223
a. Definition, 223
b. Properties of Estimable Functions, 224
c. Confidence Intervals, 227
d. What Functions Are Estimable?, 228
e. Linearly Independent Estimable Functions, 229
f. Testing for Estimability, 229
g. General Expressions, 233
5. The General Linear Hypothesis, 236
a. Testable Hypotheses, 236
b. Testing Testable Hypothesis, 237
c. The Hypothesis K'b = 0, 240
d. Non-testable Hypothesis, 241
e. Checking for Testability, 243
f. Some Examples of Testing Hypothesis, 245
g. Independent and Orthogonal Contrasts, 248
h. Examples of Orthogonal Contrasts, 250
6. Restricted Models, 255
a. Restrictions Involving Estimable Functions, 257
b. Restrictions Involving Non-estimable Functions, 259
c. Stochastic Constraints, 260
7. The "Usual Constraints", 264
a. Limitations on Constraints, 266
b. Constraints of the Form bº i = 0, 266
c. Procedure for Deriving bº and G, 269
d. Restrictions on the Model, 270
e. Illustrative Examples of Results in Subsections a-d, 272
8. Generalizations, 276
a. Non-singular V, 277
b. Singular V, 277
9. An Example, 280
10. Summary, 283
11. Exercises, 283
6. Two Elementary Models 287
1. Summary of the General Results, 288
2. The One-Way Classification, 291
a. The Model, 291
b. The Normal Equations, 294
c. Solving the Normal Equations, 294
d. Analysis of Variance, 296
e. Estimable Functions, 299
f. Tests of Linear Hypotheses, 304
g. Independent and Orthogonal Contrasts, 308
h. Models that Include Restrictions, 310
i. Balanced Data, 312
3. Reductions in Sums of Squares, 313
a. The R( ) Notation, 313
b. Analyses of Variance, 314
c. Tests of Hypotheses, 315
4. Multiple Comparisons, 316
5. Robustness of Analysis of Variance to Assumptions, 321
a. Non-normality of the Error, 321
b. Unequal Variances, 325
c. Non-independent Observations, 330
6. The Two-Way Nested Classification, 331
a. Model, 332
b. Normal Equations, 332
c. Solving the Normal Equations, 333
d. Analysis of Variance, 334
e. Estimable Functions, 336
f. Tests of Hypothesis, 337
g. Models that Include Restrictions, 339
h. Balanced Data, 339
7. Normal Equations for Design Models, 340
8. A Few Computer Outputs, 341
9. Exercises, 343
7. The Two-Way Crossed Classification 347
1. The Two-Way Classification Without Interaction, 347
a. Model, 348
b. Normal Equations, 349
c. Solving the Normal Equations, 350
d. Absorbing Equations, 352
e. Analyses of Variance, 356
f. Estimable Functions, 368
g. Tests of Hypothesis, 370
h. Models that Include Restrictions, 373
i. Balanced Data, 374
2. The Two-Way Classification with Interaction, 380
a. Model, 381
b. Normal Equations, 383
c. Solving the Normal Equations, 384
d. Analysis of Variance, 385
(i) Basic Calculations, 385
(ii) Fitting Different Models, 389
(iii) Computational Alternatives, 395
(iv) Interpretation of Results, 397
(v) Fitting Main Effects Before Interaction, 397
e. Estimable Functions, 398
f. Tests of Hypotheses, 403
g. Models that Include Restrictions, 413
h. All Cells Filled, 414
i. Balanced Data, 415
3. Interpretation of Hypotheses, 420
4. Connectedness, 422
5. The µij Models, 427
6. Exercises, 429
8. Some Other Analyses 437
1. Large-Scale Survey-Type Data, 437
a. Example, 438
b. Fitting a Linear Model, 438
c. Main-Effects-Only Models, 440
d. Stepwise Fitting, 442
e. Connectedness, 442
f. The µij-models, 443
2. Covariance, 445
a. A General Formulation, 446
b. The One-Way Classification, 454
c. The Two-Way Classification (With Interaction), 470
3. Data Having All Cells Filled, 474
a. Estimating Missing Observations, 475
b. Setting Data Aside, 478
c. Analysis of Means, 479
d. Separate Analyses, 487
4. Exercises, 487
9. Introduction to Variance Components 493
1. Fixed and Random Models, 493
a. A Fixed-Effects Model, 494
b. A Random-Effects Model, 494
c. Other Examples, 496
2. Mixed Models, 497
3. Fixed or Random, 499
4. Finite Populations, 500
5. Introduction to Estimation, 500
a. Variance Matrix Structures, 501
b. Analyses of Variance, 502
c. Estimation, 504
6. Rules for Balanced Data, 507
a. Establishing Analysis of Variance Tables, 507
b. Calculating Sums of Squares, 510
c. Expected Values of Mean Squares, E(MS), 510
7. The Two-Way Classification, 512
a. The Fixed-Effects Model, 515
b. Random-Effects Model, 518
c. The Mixed Model, 521
8. Estimating Variance Components from Balanced Data, 526
a. Unbiasedness and Minimum Variance, 527
b. Negative Estimates, 528
9. Normality Assumptions, 530
a. Distribution of Mean Squares, 530
b. Distribution of Estimators, 532
c. Tests of Hypothesis, 533
d. Confidence Intervals, 536
e. Probability of Negative Estimates, 538
f. Sampling Variances of Estimators, 539
10. Other Ways to Estimate Variance Components, 542
a. Maximum Likelihood Methods, 542
b. The MINQUE, 545
c. Bayes Estimation, 554
11. Exercises, 557
10. Methods of Estimating Variance Components from Unbalanced Data 563
1. Expectations of Quadratic Forms, 563
a. Fixed-Effects Models, 564
b. Mixed Models, 565
c. Random-Effects Models, 566
d. Applications, 566
2. Analysis of Variance Method (Henderson's Method 1), 567
a. Model and Notation, 567
b. Analogous Sums of Squares, 568
c. Expectations, 569
d. Sampling Variances of Estimators, 577
3. Adjusting for Bias in Mixed Models, 588
a. General Method, 588
b. A Simplification, 588
c. A Special Case: Henderson's Method 2, 589
4. Fitting Constants Method (Henderson's Method 3), 590
a. General Properties, 590
b. The Two-Way Classification, 592
c. Too Many Equations, 595
d. Mixed Models, 597
e. Sampling Variances of Estimators, 597
5. Analysis of Means Methods, 598
6. Symmetric Sums Methods, 599
7. Infinitely Many Quadratics, 602
8. Maximum Likelihood for Mixed Models, 605
a. Estimating Fixed Effects, 606
b. Fixed Effects and Variance Components, 611
c. Large Sample Variances, 613
9. Mixed Models Having One Random Factor, 614
10. Best Quadratic Unbiased Estimation, 620
a. The Method of Townsend and Searle (1971) for a Zero Mean, 620
b. The Method of Swallow and Searle (1978) for a Non-Zero Mean, 622
11. Shrinkage Estimation of Regression Parameters and Variance Components, 626
a. Shrinkage Estimators, 626
b. The James-Stein Estimator, 627
c. Stein's Estimator of the Variance, 627
d. A Shrinkage Estimator of Variance Components, 628
12. Exercises, 630
References 633
Author Index 645
Subject Index 649
PREFACE
I was both honored and humbled when, in November 2013, Stephen Quigley, then an associate publisher for John Wiley & Sons, now retired, asked me whether I would like to prepare a second edition of Searle's Linear Models. The first edition was my textbook when I studied linear models as a graduate student in statistics at the University of Rochester during the seventies. It has served me well as an important reference since then. I hope that this edition represents an improvement in the content, presentation, and timeliness of this well-respected classic. Indeed, Linear Models is a basic and very important tool for statistical analysis. The content and the level of this new edition is the same as the first edition with a number of additions and enhancements. There are also a few changes.
As pointed out in the first edition preface, the prerequisites for this book include a semester of matrix algebra and a year of statistical methods. In addition, knowledge of some of the topics in Gruber (2014) and Searle (2006) would be helpful.
The first edition had 11 chapters. The chapters in the new edition correspond to those in the first edition with a few changes and some additions. A short introductory chapter, Introduction and Overview is added at the beginning. This chapter gives a brief overview of what the entire book is about. Hopefully, this will give the reader some insight as to why some of the topics are taken up where they are. Chapters 1-10 are with additions and enhancements, the same as those of the first edition. Chapter 11, a list of formulae for estimating variance components in an unbalanced model is exactly as it was presented in the first edition. There are no changes in Chapter 11. This Chapter is available at the book's webpage www.wiley.com\go\Searle\LinearModels2E.
Here is howthe content of Chapters 1-10 has been changed, added to, or enhanced.
In Chapter 1, the following topics have been added to the discussion of generalized inverses:
- The singular value decomposition;
- A representation of the Moore-Penrose inverse in terms of the singular value decomposition;
- A representation of any generalized inverse in terms of the Moore-Penrose inverse;
- A discussion of reflexive, least-square generalized, and minimum norm generalized inverses with an explanation of the relationships between them and the Moore-Penrose inverse.
The content of Chapter 2 is the same as that of the first edition with the omission of the section on singular normal distributions. Here, the reference is given to the first edition.
Chapter 3 has a number of additions and enhancements. Reviewers of the first edition claimed that the Gauss-Markov theorem was not discussed there. Actually, it was but not noted as such. I gave a formal statement and proof of this important result. I also gave an extension of the Gauss-Markov theorem to models where the parameterswere random variables. This leads to a discussion of ridge-type estimators.
As was the case in the first edition, many of the numerical illustrations in Chapters 3-8 use hypothetical data. However, throughout the rest of the book, I have added some illustrative examples using real or simulated data collected from various sources. I have given SAS and R output for these data sets. In most cases, I did include the code. The advent of personal computers since the writing of the first edition makes this more relevant and easier to do than in 1971. When presenting hypothesis tests and confidence intervals, the notion of using p-values, as well as acceptance or rejection regions, was used. I made mention of how to calculate these values or obtain critical regions using graphing calculators like the TI 83 or 84. These enhancements were also made in the later chapters where appropriate.
Chapter 4 was pretty much the same as in the first edition with some changes in the exercises to make them more specific as opposed to being open-ended.
In addition to some of the enhancements mentioned for Chapter 3, Chapter 5 contains the following additional items:
- Alternative definitions of estimable functions in terms of the singular value decomposition;
- A formal statement and proof of the Gauss-Markov theorem for the non-full rank model using a Lagrange multiplier argument;
- Specific examples using numbers in matrices of tests for estimability;
- An example of how for hypothesis involving non-estimable functions using least-square estimators derived from different generalized inverses will yield different F-statistics.
In addition to the material of the first edition, Chapter 6 contains the following new items:
- A few examples for the balanced case;
- Some examples with either small "live" or simulated data sets;
- A discussion of and examples of multiple comparisons, in particular Bonferonni and Scheffe simultaneous confidence intervals;
- A discussion of the robustness of assumptions of normality, equal variances, and independent observations in analysis of variance;
- Some non-parametric procedures for dealing with non-normal data;
- A few examples illustrating the use of the computer packages SAS and R.
These items are also given for the two-way models that are considered in Chapter 7. In addition, an explanation of the difference between the Type I and Type III sum of squares in SAS is included. This is of particular importance for unbalanced data.
Chapter 8 presents three topics-missing values, analysis of covariance, and largescale survey data. The second edition contains some numerical examples to illustrate why doing analysis considering covariates is important.
Chapter 9, in addition to the material in the first edition:
- Illustrates "brute force" methods for computing expected mean squares in random and mixed models;
- Clarifies and gives examples of tests of significance for variance components;
- Presents and gives examples of the MINQUE, Bayes, and restricted Bayes estimator for estimating the variance components.
New in Chapter 10 are:
- More discussion and examples of the MINQUE;
- The connection between the maximum likelihood method and the best linear unbiased predictor.
- Shrinkage methods for the estimation of variance components.
The references are listed after Chapter 10. They are all cited in the text. Many of them are new to the second edition and of course more recent. The format of the bibliography is the same as that of the first edition.
Chapter 11, the statistical tables from the first edition, and the answers to selected exercises are contained on the web page www.wiley.com\go\Searle\LinearModels2E. A solutions manual containing the solutions to all of the exercises is available to instructors using this book as a text for their course.
There are about 15% more exercises than in the first edition. Many of the exercises are those of the first edition, in some cases reworded to make them clearer and less open-ended.
The second edition contains more numerical examples and exercises than the first edition. Numerical exercises appear before the theoretical ones at the end of each chapter.
For the most part, notations are the same as those in the first edition. Letters in equations are italic. Vectors and matrices are boldfaced. With hopes of making reading easier, many of the longer sentences have been broken down to two or three simpler sentences. Sections containing material not in the first edition has been put in between the original sections where I thought it appropriate.
The method of numbering sections is the same as in the first edition using Arabic numbers for sections, lower case letters for sub-sections, and lower case roman numerals for sub-sub sections. Unlike the first edition, examples are numbered within each chapter as Example 1, Example 2, Example 3, etc., the numbering starting fresh in each new chapter. Examples end with , formal proofs with . Formal definitions are in boxes.
I hope that I have created a second edition of this great work that is timely and reader-friendly. I appreciate any comments the readers may have about this.
A project like this never gets done without the help of other people. There were several members of the staff of John Wiley & Sons whom I would like to thank for help in various ways. My sincere thanks to Stephen H. Quigley, former Associate Publisher, for suggesting this project and for his helpful guidance during its early stages. I hope that he is enjoying his retirement. I would also like to express my gratitude to his successor Jon Gurstelle for his help in improving the timeliness of this work. I am grateful to Sari Friedman and Allison McGinniss and the production staff atWiley for theirwork dealingwith the final manuscript. In addition, Iwould like to thank the production editors Danielle LaCourciere ofWiley and Suresh Srinivasan of Aptara for the work on copyediting. Thanks are also due to Kathleen Pagliaro of Wiley for her work on the cover. The efforts of these people certainly made this a better book.
I would like to thank my teachers at the University of Rochester Reuben Gabriel, Govind Mudolkhar, and Poduri Rao for introducing me to linear models.
Special thanks go to Michal Barbosu, Head of the School of Mathematical Sciences at the Rochester Institute of Technology for helping to make SAS software available. I am grateful to my colleague Nathan Cahill and his graduate student Tommy Keane for help in the use of R statistical...
