Robust Correlation
Description
Reviews / Votes
"This book can be used as a reference book for professional statisticians and users of statistical methods. It can also serve as a graduate level textbook for a special topic course on robust correlation" Yuehua Wu, MathSciNet, Aug 2017More details
Other editions
Additional editions
Persons
Content
Preface xv
Acknowledgements xvii
About the Companion Website xix
1 Introduction 1
1.1 Historical Remarks 1
1.2 Ontological Remarks 4
1.2.1 Forms of data representation 5
1.2.2 Types of data statistics 5
1.2.3 Principal aims of statistical data analysis 6
1.2.4 Prior information about data distributions and related approaches to statistical data analysis 6
References 8
2 Classical Measures of Correlation 10
2.1 Preliminaries 10
2.2 Pearson's Correlation Coefficient: Definitions and Interpretations 12
2.2.1 Introductory remarks 13
2.2.2 Correlation via regression 13
2.2.3 Correlation via the coefficient of determination 16
2.2.4 Correlation via the variances of the principal components 18
2.2.5 Correlation via the cosine of the angle between the variable vectors 21
2.2.6 Correlation via the ratio of two means 22
2.2.7 Pearson's correlation coefficient between random events 23
2.3 Nonparametric Measures of Correlation 24
2.3.1 Introductory remarks 24
2.3.2 The quadrant correlation coefficient 26
2.3.3 The Spearman rank correlation coefficient 27
2.3.4 The Kendall ¿¿¿¿-rank correlation coefficient 28
2.3.5 Concluding remark 29
2.4 Informational Measures of Correlation 29
2.5 Summary 31
References 31
3 Robust Estimation of Location 33
3.1 Preliminaries 33
3.2 Huber's Minimax Approach 35
3.2.1 Introductory remarks 35
3.2.2 Minimax variance M-estimates of location 36
3.2.3 Minimax bias M-estimates of location 43
3.2.4 L-estimates of location 44
3.2.5 R-estimates of location 45
3.2.6 The relations between M-, L- and R-estimates of location 46
3.2.7 Concluding remarks 47
3.3 Hampel's Approach Based on Influence Functions 47
3.3.1 Introductory remarks 47
3.3.2 Sensitivity curve 47
3.3.3 Influence function and its properties 49
3.3.4 Local measures of robustness 51
3.3.5 B- and V-robustness 52
3.3.6 Global measure of robustness: the breakdown point 52
3.3.7 Redescending M-estimates 53
3.3.8 Concluding remark 56
3.4 Robust Estimation of Location: A Sequel 56
3.4.1 Introductory remarks 56
3.4.2 Huber's minimax variance approach in distribution density models of a non-neighborhood nature 57
3.4.3 Robust estimation of location in distribution models with a bounded variance 62
3.4.4 On the robustness of robust solutions: stability of least informative distributions 69
3.4.5 Concluding remark 73
3.5 Stable Estimation 73
3.5.1 Introductory remarks 73
3.5.2 Variance sensitivity 74
3.5.3 Estimation stability 76
3.5.4 Robustness of stable estimates 78
3.5.5 Maximin stable redescending M-estimates 83
3.5.6 Concluding remarks 84
3.6 Robustness Versus Gaussianity 85
3.6.1 Introductory remarks 85
3.6.2 Derivations of the Gaussian distribution 87
3.6.3 Properties of the Gaussian distribution 92
3.6.4 Huber's minimax approach and Gaussianity 100
3.6.5 Concluding remarks 101
3.7 Summary 102
References 102
4 Robust Estimation of Scale 107
4.1 Preliminaries 107
4.1.1 Introductory remarks 107
4.1.2 Estimation of scale in data analysis 108
4.1.3 Measures of scale defined by functionals 110
4.2 M- and L-Estimates of Scale 111
4.2.1 M-estimates of scale 111
4.2.2 L-estimates of scale 115
4.3 Huber Minimax Variance Estimates of Scale 116
4.3.1 Introductory remarks 116
4.3.2 The least informative distribution 117
4.3.3 Minimax variance M- and L-estimates of scale 118
4.4 Highly Efficient Robust Estimates of Scale 119
4.4.1 Introductory remarks 119
4.4.2 The median of absolute deviations and its properties 120
4.4.3 The quartile of pair-wise absolute differences Qn estimate and its properties 121
4.4.4 M-estimate approximations to the Qn estimate: MQ¿¿¿¿n, FQ¿¿¿¿n , and FQn estimates of scale 122
4.5 Monte Carlo Experiment 130
4.5.1 A remark on the Monte Carlo experiment accuracy 131
4.5.2 Monte Carlo experiment: distribution models 131
4.5.3 Monte Carlo experiment: estimates of scale 132
4.5.4 Monte Carlo experiment: characteristics of performance 133
4.5.5 Monte Carlo experiment: results 134
4.5.6 Monte Carlo experiment: discussion 136
4.5.7 Concluding remarks 138
4.6 Summary 138
References 139
5 Robust Estimation of Correlation Coefficients 140
5.1 Preliminaries 140
5.2 Main Groups of Robust Estimates of the Correlation Coefficient 141
5.2.1 Introductory remarks 141
5.2.2 Direct robust counterparts of Pearson's correlation coefficient 142
5.2.3 Robust correlation via nonparametric measures of correlation 143
5.2.4 Robust correlation via robust regression 143
5.2.5 Robust correlation via robust principal component variances 145
5.2.6 Robust correlation via two-stage procedures 147
5.2.7 Concluding remarks 147
5.3 Asymptotic Properties of the Classical Estimates of the Correlation Coefficient 148
5.3.1 Pearson's sample correlation coefficient 148
5.3.2 The maximum likelihood estimate of the correlation coefficient at the normal 149
5.4 Asymptotic Properties of Nonparametric Estimates of Correlation 151
5.4.1 Introductory remarks 151
5.4.2 The quadrant correlation coefficient 152
5.4.3 The Kendall rank correlation coefficient 152
5.4.4 The Spearman rank correlation coefficient 153
5.5 Bivariate Independent Component Distributions 155
5.5.1 Definition and properties 155
5.5.2 Independent component and Tukey gross-error distribution models 156
5.6 Robust Estimates of the Correlation Coefficient Based on Principal Component Variances 158
5.7 Robust Minimax Bias and Variance Estimates of the Correlation Coefficient 161
5.7.1 Introductory remarks 161
5.7.2 Minimax property 162
5.7.3 Concluding remarks 163
5.8 Robust Correlation via Highly Efficient Robust Estimates of Scale 163
5.8.1 Introductory remarks 163
5.8.2 Asymptotic bias and variance of generalized robust estimates of the correlation coefficient 164
5.8.3 Concluding remarks 165
5.9 Robust M-Estimates of the Correlation Coefficient in Independent Component Distribution Models 165
5.9.1 Introductory remarks 165
5.9.2 The maximum likelihood estimate of the correlation coefficient in independent component distribution models 165
5.9.3 M-estimates of the correlation coefficient 166
5.9.4 Asymptotic variance of M-estimators 166
5.9.5 Minimax variance M-estimates of the correlation coefficient 167
5.9.6 Concluding remarks 168
5.10 Monte Carlo Performance Evaluation 168
5.10.1 Introductory remarks 168
5.10.2 Monte Carlo experiment set-up 168
5.10.3 Discussion 171
5.10.4 Concluding remarks 173
5.11 Robust Stable Radical M-Estimate of the Correlation Coefficient of the Bivariate Normal Distribution 173
5.11.1 Introductory remarks 173
5.11.2 Asymptotic characteristics of the stable radical estimate of the correlation coefficient 174
5.11.3 Concluding remarks 175
5.12 Summary 176
References 176
6 Classical Measures of Multivariate Correlation 178
6.1 Preliminaries 178
6.2 Covariance Matrix and Correlation Matrix 179
6.3 Sample Mean Vector and Sample Covariance Matrix 181
6.4 Families of Multivariate Distributions 182
6.4.1 Construction of multivariate location-scatter models 182
6.4.2 Multivariate symmetrical distributions 183
6.4.3 Multivariate normal distribution 184
6.4.4 Multivariate elliptical distributions 184
6.4.5 Independent component model 186
6.4.6 Copula models 186
6.5 Asymptotic Behavior of Sample Covariance Matrix and Sample Correlation Matrix 187
6.6 First Uses of Covariance and Correlation Matrices 189
6.7 Working with the Covariance Matrix-Principal Component Analysis 191
6.7.1 Principal variables 191
6.7.2 Interpretation of principal components 193
6.7.3 Asymptotic behavior of the eigenvectors and eigenvalues 194
6.8 Working with Correlations-Canonical Correlation Analysis 195
6.8.1 Canonical variates and canonical correlations 195
6.8.2 Testing for independence between subvectors 197
6.9 Conditionally Uncorrelated Components 199
6.10 Summary 200
References 200
7 Robust Estimation of Scatter and Correlation Matrices 202
7.1 Preliminaries 202
7.2 Multivariate Location and Scatter Functionals 202
7.3 Influence Functions and Asymptotics 205
7.4 M-functionals for Location and Scatter 208
7.5 Breakdown Point 210
7.6 Use of Robust Scatter Matrices 211
7.6.1 Ellipticity assumption 211
7.6.2 Robust correlation matrices 212
7.6.3 Principal component analysis 212
7.6.4 Canonical correlation analysis 213
7.7 Further Uses of Location and Scatter Functionals 213
7.8 Summary 215
References 215
8 Nonparametric Measures of Multivariate Correlation 217
8.1 Preliminaries 217
8.2 Univariate Signs and Ranks 218
8.3 Marginal Signs and Ranks 220
8.4 Spatial Signs and Ranks 222
8.5 Affine Equivariant Signs and Ranks 226
8.6 Summary 229
References 230
9 Applications to Exploratory Data Analysis: Detection of Outliers 231
9.1 Preliminaries 231
9.2 State of the Art 232
9.2.1 Univariate boxplots 232
9.2.2 Bivariate boxplots 234
9.3 Problem Setting 237
9.4 A New Measure of Outlier Detection Performance 239
9.4.1 Introductory remarks 240
9.4.2 H-mean: motivation, definition and properties 241
9.5 Robust Versions of the Tukey Boxplot with Their Application to Detection of Outliers 243
9.5.1 Data generation and performance measure 243
9.5.2 Scale and shift contamination 243
9.5.3 Real-life data results 244
9.5.4 Concluding remarks 245
9.6 Robust Bivariate Boxplots and Their Performance Evaluation 245
9.6.1 Bivariate FQ-boxplot 245
9.6.2 Bivariate FQ-boxplot performance 247
9.6.3 Measuring the elliptical deviation from the convex hull 249
9.7 Summary 253
References 253
10 Applications to Time Series Analysis: Robust Spectrum Estimation 255
10.1 Preliminaries 255
10.2 Classical Estimation of a Power Spectrum 256
10.2.1 Introductory remarks 256
10.2.2 Classical nonparametric estimation of a power spectrum 258
10.2.3 Parametric estimation of a power spectrum 259
10.3 Robust Estimation of a Power Spectrum 259
10.3.1 Introductory remarks 259
10.3.2 Robust analogs of the discrete Fourier transform 261
10.3.3 Robust nonparametric estimation 262
10.3.4 Robust estimation of power spectrum through the Yule-Walker equations 263
10.3.5 Robust estimation through robust filtering 263
10.4 Performance Evaluation 264
10.4.1 Robustness of the median Fourier transform power spectra 264
10.4.2 Additive outlier contamination model 264
10.4.3 Disorder contamination model 264
10.4.4 Concluding remarks 270
10.5 Summary 270
References 270
11 Applications to Signal Processing: Robust Detection 272
11.1 Preliminaries 272
11.1.1 Classical approach to detection 272
11.1.2 Robust minimax approach to hypothesis testing 273
11.1.3 Asymptotically optimal robust detection of a weak signal 274
11.2 Robust Minimax Detection Based on a Distance Rule 275
11.2.1 Introductory remarks 275
11.2.2 Asymptotic robust minimax detection of a known constant signal with the ¿¿¿¿-distance rule 276
11.2.3 Detection performance in asymptotics and on finite samples 278
11.2.4 Concluding remarks 283
11.3 Robust Detection of a Weak Signal with Redescending M-Estimates 285
11.3.1 Introductory remarks 285
11.3.2 Detection error sensitivity and stability 287
11.3.3 Performance evaluation: a comparative study 289
11.3.4 Concluding remarks 291
11.4 A Unified Neyman-Pearson Detection of Weak Signals in a Fusion Model with Fading Channels and Non-Gaussian Noises 296
11.4.1 Introductory remarks 296
11.4.2 Problem setting-an asymptotic fusion rule 298
11.4.3 Asymptotic performance analysis 299
11.4.4 Numerical results 303
11.4.5 Concluding remarks 305
11.5 Summary 306
References 306
12 Final Remarks 308
12.1 Points of Growth: Open Problems in Multivariate Statistics 308
12.2 Points of Growth: Open Problems in Applications 309
Index 311
Chapter 1
Introduction
This book is most about correlation, association and partially about regression, i.e., about those areas of science where the dependencies between random variables that mathematically describe the relations between observed phenomena and associated with them features are studied. Evidently, these concepts and terms firstly appeared in applied sciences, not in mathematics. Below we briefly overview the historical aspects of the considered concepts.
1.1 Historical Remarks
The word "correlation" is of late Latin origin meaning "association", "connection", "correspondence", "interdependence", "relationship", but relationship not in the conventional for that time deterministic functional form.
The term "correlation" was introduced into science by a French naturalist Georges Cuvier (1769-1832), one of the major figures in natural sciences in the early 19th century, who had founded paleontology and comparative anatomy. Cuvier discovered and studied the relationships between the parts of animals, between the structure of animals and their mode of existence, between the species of animals and plants, and many others. This experience made him establish the general principles of "the correlation of parts" and of "the functional correlation" (Rudwick 1997):
Today comparative anatomy has reached such a point of perfection that, after inspecting a single bone, one can often determine the class, and sometimes even the genus of the animal to which it belonged, above all if that bone belonged to the head or the limbs. . This is because the number, direction, and shape of the bones that compose each part of an animal's body are always in a necessary relation to all the other parts, in such a way that - up to a point - one can infer the whole from any one of them and vice versa.
From Cuvier to Galton, correlation had been understood as a qualitatively described relationship, not deterministic but of a statistical nature, however observed at that time within a rather narrow area of phenomena.
The notion of regression is connected with the great names of Laplace, Legendre, Gauss, and Galton (1885), who coined this term. Laplace (1799) was the first to propose a method for processing the astronomical data, namely, the least absolute values method. Legendre (1805) and Gauss (1809) independently of each other introduced the least squares method.
Francis Galton (1822-1911), a British anthropologist, biologist, psychologist, andmeteorologist, understood that correlation is the interrelationship in average between any random variables (Galton 1888):
Two variable organs are said to be co-related when the variation of the one is accompanied on the average by more or less variation of the other, and in the same direction.. It is easy to see that co-relation must be the consequence of the variations of the two organs being partly due to common cause.. If they were in no respect due to common causes, the co-relation would be nil.
Correlation analysis (this term also was coined by Galton) deals with estimation of the value of correlation by number indexes or coefficients.
Similarly to Cuvier, Galton introduced regression dependence observing live nature, in particular, processing the heredity and sweet peas data (Galton 1894). Regression characterizes the correlation dependence between random variables functionally in average. Studying the sizes of sweet peas beans, he noticed that the offspring seeds did not reveal the tendency to reproduce the size of their parents being closer to the population mean than them. Namely, the seeds were smaller than their parents in the case of large parent sizes, and vice versa. Galton called this dependence regression, for the reverse changes had been observed: firstly, he used the term "the law of reversion". Further studies showed that on average the offspring regression to the population mean was proportional to the parent deviations from it - this allowed the observed dependence to be described using the linear function. The similar linear regression is described by Galton as a result of processing the heights of 930 adult children and their 205 parents (Galton 1894).
The term "regression" became popular, and now it is used in the case of functional dependencies in average between any random variables. Using modern terminology, we may say that Galton considered the slope of the simple linear regression line as a measure of correlation (Galton 1888):
Let the deviation of the subject [in units of the probably error, ], whichever of the two variables may be taken in that capacity; and let be the corresponding deviations of the relative, and let the mean of these be . Then we find: (1) that for all values of ; (2) that is the same, whichever of the two variables is taken for the subject; (3) that is always less than 1; (4) that measures the closeness of co-relation.
Now we briefly comment on the above-mentioned properties (1)-(4): the first is just the simple linear regression equation between the standardized variables and ; the second means that the co-relation is symmetric with regard to the variables and ; the third and fourth show that Galton had not yet recognized the idea of negative correlation: stating that could not be greater than 1, he evidently understood as a positive measure of "co-relation". Originally stood for the regression slope, and that is really so for the standardized variables; Galton perceived the correlation coefficient as a scale invariant regression slope.
Galton contributed much to science studying the problems of heredity of qualitative and quantitative features. They were numerically examined by Galton on the basis of the concept of correlation. Until the present, the data on demography, heredity, and sociology collected by Galton with the corresponding numerical examples of correlations computed are used.
Karl Pearson (1857-1936), a British mathematician, statistician, biologist, and philosopher, had written out the explicit formulas for the population product-moment correlation coefficient (Pearson 1895)
1.1and its sample version
1.2(here and are the sample means of the observations and of random variables and ). However, Pearson did not definitely distinguish the population and sample versions of the correlation coefficient, as it is commonly done at present.
Thus, on the one hand, the sample correlation coefficient is a statistical counterpart of the correlation coefficient of a bivariate distribution, where , , and are the variances and the covariance of the random variables and , respectively.
On the other hand, it is an efficient maximum likelihood estimate of the correlation coefficient of the bivariate normal distribution (Kendall and Stuart 1963) with density
1.3where , , , .
Galton (1888) derived the bivariate normal distribution (1.3), and he was the first who used it to scatter the frequencies of children's stature and parents' stature. Pearson noted that "in 1888 Galton had completed the theory of bivariate normal correlation" (Pearson 1920).
Like Galton, Auguste Bravais (1846), a French naval officer and astronomer, came very near to the definition (1.1) when he called one parameter of the bivariate normal distribution "une correlation", but he did not recognize it as a measure of the interrelationship between variables. However, "his work in Pearson's hands proved useful in framing formal approaches in those areas" (Stigler 1986).
Pearson's formulas (1.1) and (1.2) proved to be fruitful for studying dependencies: correlation analysis and most of multivariate statistical analysis tools are based on the pair-wise Pearson correlations; we may also add the correlation and spectral theories of stochastic processes, etc.
Since the time Pearson introduced the sample correlation coefficient (1.2), many other measures of correlation have been used aiming at estimation of the closeness of interrelationship (the coefficients of association, determination, contingency, etc.). Some of them were proposed by Karl Pearson (1920).
It would not be out of place to note the contributions to correlation analysis of the other British statisticians.
Ronald Fisher (1890-1962) is one of the creators of mathematical statistics. In particular, he is the originator of the analysis of variance and together with Karl Pearson he stands at the beginning of the theory of hypothesis testing. He introduced the notion of a sufficient statistic and proposed the maximum likelihood method (Fisher 1922). Fisher also payed much attention to correlation analysis: his tools for verifying the significance of correlation under the normal law are used until now.
George Yule (1871-1951) is a prominent statistician of the first half of the 20th century. He contributed much to the statistical theories of regression, correlation (Yule's coefficient of contingency between random events), and spectral analysis.
Maurice Kendall (1907-1983) is one of the creators of nonparametric statistics, in particular, of the nonparametric...
