Permutation Tests for Complex Data
Theory, Applications and Software
2nd Edition
Published on 14. August 2025
Book
Hardback
560 pages
978-1-119-43823-6 (ISBN)
Description
TIMELY UPDATE OF A POPULAR EDITION ON PERMUTATION TESTING WITH NUMEROUS CASE STUDIES INCLUDED THROUGHOUT
The newly revised and updated Second Edition of Permutation Tests for Complex Data describes permutation tests from the point of view of experimental design, with methodological details and illustrating the process of devising an appropriate permutation test through case studies. In addition to the text, this book includes two open source packages for permutation tests in Python and R which include a comprehensive code base to implement common permutation tests as well as code to implement each of the book's case studies.
The focus of this book is the permutation approach to a variety of univariate and multivariate problems of hypothesis testing in a typical nonparametric framework. The book examines the most up-to-date methodologies of univariate and multivariate permutation testing, includes real case studies from both experimental and observational studies, and presents and discusses solutions to the most important and frequently encountered real problems in multivariate analyses.
Written by two highly qualified authors in the field of nonparametrics and applied statistics, Permutation Tests for Complex Data includes information on sample topics including:
Theory of one-dimensional and multi-dimensional permutation tests, covering test statistics, arguments for selecting permutation tests, and examples of one-sample and multi-sample problems
Multiplicity control and closed testing, covering raw and adjusted p-values, the MinP Bonferroni-Holm procedure, and weighted methods for controlling FWE and FDR
Multivariate categorical variables, covering stochastic ordering, tests on moments for ordered variables, and heterogeneity comparisons
NPC tests for survival analysis and shape analysis, covering analysis of PERC data and analysis with correlated landmarks
Presenting a thorough overview of permutation testing with both formal description and proofs, Permutation Tests for Complex Data is an excellent introduction to permutation tests for graduate-level statistics or data science courses and will be ideal as a handbook for researchers hoping to use the open source code.
The newly revised and updated Second Edition of Permutation Tests for Complex Data describes permutation tests from the point of view of experimental design, with methodological details and illustrating the process of devising an appropriate permutation test through case studies. In addition to the text, this book includes two open source packages for permutation tests in Python and R which include a comprehensive code base to implement common permutation tests as well as code to implement each of the book's case studies.
The focus of this book is the permutation approach to a variety of univariate and multivariate problems of hypothesis testing in a typical nonparametric framework. The book examines the most up-to-date methodologies of univariate and multivariate permutation testing, includes real case studies from both experimental and observational studies, and presents and discusses solutions to the most important and frequently encountered real problems in multivariate analyses.
Written by two highly qualified authors in the field of nonparametrics and applied statistics, Permutation Tests for Complex Data includes information on sample topics including:
Theory of one-dimensional and multi-dimensional permutation tests, covering test statistics, arguments for selecting permutation tests, and examples of one-sample and multi-sample problems
Multiplicity control and closed testing, covering raw and adjusted p-values, the MinP Bonferroni-Holm procedure, and weighted methods for controlling FWE and FDR
Multivariate categorical variables, covering stochastic ordering, tests on moments for ordered variables, and heterogeneity comparisons
NPC tests for survival analysis and shape analysis, covering analysis of PERC data and analysis with correlated landmarks
Presenting a thorough overview of permutation testing with both formal description and proofs, Permutation Tests for Complex Data is an excellent introduction to permutation tests for graduate-level statistics or data science courses and will be ideal as a handbook for researchers hoping to use the open source code.
More details
Series
Edition
2nd edition
Language
English
Place of publication
New York
United States
Target group
Professional and scholarly
Product notice
sewn/stitched
Cloth over boards
Dimensions
Height: 230 mm
Width: 154 mm
Thickness: 34 mm
Weight
728 gr
ISBN-13
978-1-119-43823-6 (9781119438236)
Copyright in bibliographic data and cover images is held by Nielsen Book Services Limited or by the publishers or by their respective licensors: all rights reserved.
Schweitzer Classification
Other editions
Additional editions
Fortunato Pesarin | Luigi Salmaso
Permutation Tests for Complex Data
Theory, Applications and Software
E-Book
07/2025
2nd Edition
Wiley
€113.99
Available for download
Fortunato Pesarin | Luigi Salmaso
Permutation Tests for Complex Data
Theory, Applications and Software
E-Book
07/2025
2nd Edition
Wiley
€113.99
Available for download
Persons
FORTUNATO PESARIN is Emeritus Professor at the Department of Statistical Sciences, University of Padova, Italy.
LUIGI SALMASO is Professor of Statistics at the Department of Management and Engineering, University of Padova, Italy.
LUIGI SALMASO is Professor of Statistics at the Department of Management and Engineering, University of Padova, Italy.
Content
Preface xvii
Acknowledgments xxi
Acronyms xxiii
Notation xxv
About the Companion Website xxix
1 Introduction 1
1.1 On Permutation Analysis 1
1.2 Basic Notation 6
1.3 The Permutation - Conditional - Testing Principle 8
1.3.1 Nonparametric Family of Distributions 9
1.3.2 The Permutation - Conditional - Testing Principle 10
1.4 Permutation Approaches 13
1.5 When and Why Conditioning Is Appropriate 15
1.6 Randomization and Permutation 18
1.7 Computational Aspects 19
1.8 A Problem with Paired Observations 20
1.8.1 Modelling Responses 21
1.8.2 Symmetry Induced by Exchangeability 23
1.8.3 Further Aspects 24
1.8.4 Student's t-Paired Solution 24
1.8.5 The Signed Rank Solution 26
1.8.6 The McNemar Solution 27
1.9 The Permutation Solution 28
1.9.1 General Aspects 28
1.9.2 The Permutation Sample Space 28
1.9.3 The Conditional Monte Carlo Method 29
1.9.3.1 An Algorithm for Inspecting Permutation Sample Spaces 29
1.9.3.2 Approximating the Permutation Distribution 31
1.9.4 Problems and Exercises 32
1.10 A Two-Sample Problem 33
1.10.1 Modelling Responses 34
1.10.2 The Student t Solution 36
1.10.3 The Permutation Solution 36
1.10.4 Rank Solutions 38
1.10.5 Fisher's Exact Probability Solution 39
1.10.6 Problems and Exercises 39
1.11 One-Way ANOVA 40
1.11.1 Modelling Responses 41
1.11.2 Permutation Solutions 42
1.11.3 Problems and Exercises 43
2 Theory of One-Dimensional Permutation Tests 45
2.1 Introduction 45
2.1.1 Notation and Basic Assumptions 45
2.1.2 The Conditional Reference Space 49
2.1.3 Conditioning on a Set of Sufficient Statistics 55
2.2 Definition of Permutation Tests 59
2.2.1 General Aspects 59
2.2.2 Randomized Permutation Tests 60
2.2.3 Non-Randomized Permutation Tests 61
2.2.4 Unbiasedness of Permutation Tests 63
2.3 Some Useful Test Statistics 65
2.4 Equivalence of Permutation Statistics 69
2.4.1 Two Examples 70
2.4.2 Problems and Exercises 72
2.5 Further Relevant Properties 74
2.5.1 The p-Value-Like Statistic 74
2.5.2 Unbiasedness of p-Value-Like Test Statistics 75
2.5.3 Weak Consistency of Permutation Tests 77
2.5.4 A CMC Algorithm for Estimating the p-Value-Like Statistic 80
2.6 Some Arguments for Choosing Permutation Tests 82
2.7 Power Functions of Permutation Tests 87
2.7.1 Definition and Algorithm for the Conditional Power 87
2.7.2 The Conditional ROC Curve 90
2.7.3 Definition and Algorithm for the Unconditional Power: Fixed Effects 90
2.7.4 Unconditional Power: Random Effects 92
2.7.5 Comments on Power Functions 92
2.8 Permutation Confidence Interval for ? 93
2.8.1 Problems and Exercises 98
2.9 Generalizing Inference from Conditional to Unconditional 99
2.10 Optimal Properties 103
2.10.1 Problems and Exercises 105
2.11 Some Further Asymptotic Properties 107
2.11.1 Introduction 107
2.11.2 Two Basic Theorems 107
2.12 Permutation Central Limit Theorems 109
2.12.1 Basic Notions 109
2.12.2 Permutation Central Limit Theorems 110
2.12.3 Problems and Exercises 113
3 Some Examples of Unidimensional Permutation Tests 117
3.1 Examples of One-Sample Problems 117
3.1.1 A Problem with Repeated Observations 125
3.1.1.1 Friedman's Rank Test 126
3.1.1.2 A Permutation Solution 126
3.1.1.3 An Example 127
3.1.2 Problems and Exercises 128
3.2 Some Examples of Multi-Sample Problems 130
3.2.1 Synchronized Permutations 139
3.3 Analysis of Ordered Categorical Variables 145
3.3.1 General Aspects 145
3.3.2 A Solution Based on Score Transformations 147
3.3.3 Typical Goodness-of-Fit Solutions 148
3.3.3.1 An Example 149
3.3.4 Extension to Non-Dominance Alternatives and C Samples 149
3.3.5 Problems and Exercises 151
4 The Nonparametric Combination Methodology 153
4.1 Introduction 153
4.1.1 General Aspects 153
4.1.2 Some Preliminary Notes 155
4.1.3 Main Assumptions and Notation 157
4.1.4 Some Comments 159
4.2 The Nonparametric Combination Methodology 160
4.2.1 Assumptions on Partial Tests 160
4.2.2 Desirable Properties of Combining Functions 162
4.2.3 A Two-Phase Algorithm for Nonparametric Combination 164
4.2.4 Some Useful Combining Functions 168
4.2.5 Why Combination Is Nonparametric 177
4.2.6 On Admissible Combining Functions 178
4.2.7 Problems and Exercises 179
4.3 Consistency, Unbiasedness, and Power of Combined Tests 182
4.3.1 Consistency 182
4.3.2 Unbiasedness 182
4.3.3 A Non-Consistent Combining Function 184
4.3.4 Power of Combined Tests 185
4.3.4.1 The Conditional Power Function 185
4.3.4.2 The Unconditional Power Function 186
4.3.5 Multivariate Conditional Confidence Region for ? 187
4.3.6 Problems and Exercises 188
4.4 Some Further Asymptotic Properties 189
4.4.1 General Conditions 189
4.4.2 Some Asymptotic Properties 190
4.5 Finite-Sample Consistency 194
4.5.1 Introduction and Motivations 194
4.5.2 Finite-Sample Consistency 195
4.5.3 Some Applications of Finite-Sample Consistency 201
4.6 Some Examples of Nonparametric Combination 207
4.6.1 Problems and Exercises 229
4.7 Comments on the Nonparametric Combination 231
4.7.1 General Comments 231
5 Multiple Testing Problems and Multiplicity Adjustment 233
5.1 Defining Raw and Adjusted p-Values 233
5.2 Controlling for Multiplicity 235
5.2.1 Multiple Comparison and Multiple Testing 235
5.2.2 Some Definitions of the Global Type I Error 236
5.3 Multiple Testing 237
5.4 The Closed Testing Approach 238
5.4.1 Closed Testing for Multiple Testing 239
5.4.2 Closed Testing Using the MinP Bonferroni-Holm Procedure 241
5.5 Weighted Methods for Controlling FWE and FDR 245
5.6 Adjusting Stepwise p-Values 247
5.6.1 Showing Biasedness of Standard p-Values for Stepwise Regression 248
5.6.2 Algorithm Description 248
5.6.3 Optimal Subset Procedures 250
6 Analysis of Multivariate Categorical Variables 251
6.1 Introduction 251
6.2 The Multivariate McNemar Test 252
6.2.1 An Extension of the Multivariate McNemar Test 255
6.3 Multivariate Goodness-of-Fit Testing for Ordered Variables 256
6.3.1 Multivariate Extension of Fisher's Exact Probability Test 258
6.4 MANOVA with Nominal Categorical Data 258
6.4.1 A Formal Description 258
6.5 Stochastic Ordering 260
6.5.1 Formal Description 260
6.5.2 Further Breaking Down of the Hypotheses 261
6.5.3 Permutation Test 261
6.6 Multifocus Analysis 264
6.6.1 General Aspects 264
6.6.2 The Multifocus Solution 264
6.6.3 An Application 267
6.7 Isotonic Inference 268
6.7.1 Introduction 268
6.7.2 Allelic Association Analysis in Genetics 270
6.7.3 Parametric Solutions 271
6.7.3.1 Chiano and Clayton's Method 271
6.7.3.2 Maximum-Likelihood Approach 272
6.7.4 Permutation Approach 273
6.8 Test on Moments for Ordered Variables 274
6.8.1 General Aspects 274
6.8.2 Score Transformations and Univariate Tests 275
6.8.3 Multivariate Extension 276
6.9 Heterogeneity Comparisons 278
6.9.1 Introduction 278
6.9.2 Tests for Comparing Heterogeneities 279
6.9.3 A Case Study in Population Genetics 281
6.10 Application to PhD Programme Evaluation 282
6.10.1 Description of the Problem 282
6.10.2 Global Satisfaction Index 284
6.10.3 Multivariate Performance Comparisons 285
7 Permutation Testing for Repeated Measurements 287
7.1 Introduction 287
7.2 Carry-Over Effects in Repeated-Measures Designs 288
7.3 Modelling Repeated Measurements 289
7.3.1 A General Additive Model 289
7.3.2 The Hypotheses of Interest 291
7.4 Testing Solutions 292
7.4.1 Solutions Using the NPC Approach 292
7.4.2 Analysis of Two-Sample Dominance Problems 294
7.4.3 Analysis of the Cross-Over (AB-BA) Design 294
7.4.4 Analysis of a Cross-Over Design with Paired Data 295
7.5 Testing for Repeated Measurements with Missing Data 296
7.5.1 General Aspects of Permutation Testing with Missing Data 297
7.5.2 Bibliographic Notes 297
7.6 On Missing Data Processes 298
7.6.1 Data Missing Completely at Random 299
7.6.2 Data Missing Not at Random 299
7.7 The Permutation Approach 300
7.7.1 Deletion, Imputation, and Intention to Treat Strategies 301
7.7.2 Breaking Down the Hypotheses 303
7.8 The Structure of Testing Problems 303
7.8.1 Hypotheses for MNAR Models 303
7.8.2 Hypotheses for MCAR Models 305
7.8.3 Permutation Structure with Missing Values 305
7.9 Permutation Analysis of Missing Values 307
7.9.1 Partitioning the Permutation Sample Space 307
7.9.2 Solution for Two-Sample MCAR Problems 308
7.9.3 Extensions to Multivariate C-Sample Problems 310
7.9.4 Extension to MNAR Models 311
7.10 'GERMINA' Data: An Example of a MNAR Model 312
7.10.1 Problem Description 313
7.10.2 The Permutation Solution 315
7.11 Multivariate Paired Observations 318
7.11.1 The Problem 318
7.12 Repeated Measures and Missing Data 319
7.12.1 An Example 320
7.13 Botulinum Data: Example 322
7.13.1 Data Description 322
7.13.2 Research Questions and Methods 323
7.14 Waterfalls Data: Example 324
7.14.1 Data Description 324
7.14.2 Research Questions and Methods 325
8 Some Stochastic Ordering Problems 327
8.1 Multivariate Ordered Alternatives 327
8.2 Testing for Umbrella Alternatives 329
8.2.1 Hypotheses and Tests in Simple Stochastic Ordering 331
8.2.2 Permutation Tests for Umbrella Alternatives 332
8.3 Tumour Growth Curves: Example 335
8.4 PERC Data: Example 336
8.4.1 Introduction 336
8.4.2 A Permutation Solution 339
9 NPC Tests for Survival Analysis 341
9.1 Introduction and Main Notation 341
9.1.1 Failure Time Distributions 341
9.1.2 Data Structure 342
9.2 Comparison of Survival Curves 343
9.3 An Overview from the Literature 345
9.3.1 Permutation Tests in Survival Analysis 347
9.4 Two NPC Tests 348
9.4.1 Breaking Down the Hypotheses 349
9.4.2 The Test Structure 350
9.4.3 NPC Test for Treatment-Independent Censoring 350
9.4.4 NPC Test for Treatment-Dependent Censoring 352
9.5 Case Study: In-Hospital Mortality 355
9.5.1 Results 355
9.6 Case Study: Esophageal Cancer 356
9.6.1 A Permutation Solution 358
9.6.2 Survival Analysis with Stratification by Propensity Score 359
9.6.3 Integrating Propensity Score and NPC Testing 361
9.7 Case Study: Comparison of Three Survival Curves 366
9.7.1 Unstratified Survival Analysis 367
9.7.2 Stratified Survival Analysis Using Propensity Score 368
10 NPC Tests in Shape Analysis 373
10.1 A Brief Overview of Statistical Shape Analysis 375
10.1.1 How to Describe Shapes 375
10.1.2 Multivariate Morphometrics 377
10.2 Inference with Shape Data 380
10.3 NPC Approach to Shape Analysis 382
10.3.1 Comparative Simulation Study 384
10.4 NPC Analysis with Correlated Landmarks 387
10.5 An Application to Mediterranean Monk Seal Skulls 390
10.5.1 Some Remarks 393
11 Case Studies 395
11.1 Autofluorescence Data 395
11.1.1 A Permutation Solution 397
11.1.2 Analysis Using R 399
11.2 Confocal Data 402
11.2.1 A Permutation Solution 404
11.2.1.1 Evaluation of Treatment Effect Using R 406
11.2.1.2 Evaluation of Time Effect Using R 409
11.2.2 Two-Way (M)ANOVA 411
11.2.2.1 Permutation Tests in Two-Way (M)ANOVA 411
11.2.2.2 Analysis Using R 413
11.3 SETIG Data 414
11.3.1 Analysis Using R 417
11.4 Logistic Regression and NPC Test for Multivariate Analysis 420
11.4.1 Lymph Node Metastases Data 420
11.4.2 Bladder Cancer Data 423
11.4.2.1 Applying NPC: Some Results 425
11.4.2.2 Analysis with Logistic Regression 425
11.4.2.3 Final Comments 427
12 Permutation Tests Over the Years 431
12.1 Kind of Data 432
12.1.1 Time-Series Data 432
12.1.2 Thick Data 432
12.1.3 Imaging Data 435
12.1.4 Survival Data 436
12.1.5 Censored Data 436
12.2 Different Models 437
12.2.1 Regression 437
12.2.2 Classification 440
12.2.3 Doe 440
12.2.4 Ranking 440
12.3 Different Tests 441
12.3.1 Test for Centrality 441
12.3.2 Test for Variability 446
12.3.3 Test for Both Centrality and Variability 446
12.3.4 ANOVA Test 447
12.3.5 Testing Relationship 447
12.3.6 NPC Tests 448
12.4 Methodological and Computational Aspects 449
12.4.1 Power 449
12.4.2 Computational Problems 452
12.4.3 Other Problems 452
Appendix A Examples with Applications in R 453
A. 1 Mult Data Example 453
A. 2 Washing Test Data 456
A. 3 Germina Data Example 460
A. 4 Botulinum Data 464
A. 5 Waterfalls Data 467
A. 6 Tumour Growth Curves 469
A. 7 PERC Data 471
A. 8 Mediterranean Monk Seal Skulls 475
Bibliography 479
Index 517
Acknowledgments xxi
Acronyms xxiii
Notation xxv
About the Companion Website xxix
1 Introduction 1
1.1 On Permutation Analysis 1
1.2 Basic Notation 6
1.3 The Permutation - Conditional - Testing Principle 8
1.3.1 Nonparametric Family of Distributions 9
1.3.2 The Permutation - Conditional - Testing Principle 10
1.4 Permutation Approaches 13
1.5 When and Why Conditioning Is Appropriate 15
1.6 Randomization and Permutation 18
1.7 Computational Aspects 19
1.8 A Problem with Paired Observations 20
1.8.1 Modelling Responses 21
1.8.2 Symmetry Induced by Exchangeability 23
1.8.3 Further Aspects 24
1.8.4 Student's t-Paired Solution 24
1.8.5 The Signed Rank Solution 26
1.8.6 The McNemar Solution 27
1.9 The Permutation Solution 28
1.9.1 General Aspects 28
1.9.2 The Permutation Sample Space 28
1.9.3 The Conditional Monte Carlo Method 29
1.9.3.1 An Algorithm for Inspecting Permutation Sample Spaces 29
1.9.3.2 Approximating the Permutation Distribution 31
1.9.4 Problems and Exercises 32
1.10 A Two-Sample Problem 33
1.10.1 Modelling Responses 34
1.10.2 The Student t Solution 36
1.10.3 The Permutation Solution 36
1.10.4 Rank Solutions 38
1.10.5 Fisher's Exact Probability Solution 39
1.10.6 Problems and Exercises 39
1.11 One-Way ANOVA 40
1.11.1 Modelling Responses 41
1.11.2 Permutation Solutions 42
1.11.3 Problems and Exercises 43
2 Theory of One-Dimensional Permutation Tests 45
2.1 Introduction 45
2.1.1 Notation and Basic Assumptions 45
2.1.2 The Conditional Reference Space 49
2.1.3 Conditioning on a Set of Sufficient Statistics 55
2.2 Definition of Permutation Tests 59
2.2.1 General Aspects 59
2.2.2 Randomized Permutation Tests 60
2.2.3 Non-Randomized Permutation Tests 61
2.2.4 Unbiasedness of Permutation Tests 63
2.3 Some Useful Test Statistics 65
2.4 Equivalence of Permutation Statistics 69
2.4.1 Two Examples 70
2.4.2 Problems and Exercises 72
2.5 Further Relevant Properties 74
2.5.1 The p-Value-Like Statistic 74
2.5.2 Unbiasedness of p-Value-Like Test Statistics 75
2.5.3 Weak Consistency of Permutation Tests 77
2.5.4 A CMC Algorithm for Estimating the p-Value-Like Statistic 80
2.6 Some Arguments for Choosing Permutation Tests 82
2.7 Power Functions of Permutation Tests 87
2.7.1 Definition and Algorithm for the Conditional Power 87
2.7.2 The Conditional ROC Curve 90
2.7.3 Definition and Algorithm for the Unconditional Power: Fixed Effects 90
2.7.4 Unconditional Power: Random Effects 92
2.7.5 Comments on Power Functions 92
2.8 Permutation Confidence Interval for ? 93
2.8.1 Problems and Exercises 98
2.9 Generalizing Inference from Conditional to Unconditional 99
2.10 Optimal Properties 103
2.10.1 Problems and Exercises 105
2.11 Some Further Asymptotic Properties 107
2.11.1 Introduction 107
2.11.2 Two Basic Theorems 107
2.12 Permutation Central Limit Theorems 109
2.12.1 Basic Notions 109
2.12.2 Permutation Central Limit Theorems 110
2.12.3 Problems and Exercises 113
3 Some Examples of Unidimensional Permutation Tests 117
3.1 Examples of One-Sample Problems 117
3.1.1 A Problem with Repeated Observations 125
3.1.1.1 Friedman's Rank Test 126
3.1.1.2 A Permutation Solution 126
3.1.1.3 An Example 127
3.1.2 Problems and Exercises 128
3.2 Some Examples of Multi-Sample Problems 130
3.2.1 Synchronized Permutations 139
3.3 Analysis of Ordered Categorical Variables 145
3.3.1 General Aspects 145
3.3.2 A Solution Based on Score Transformations 147
3.3.3 Typical Goodness-of-Fit Solutions 148
3.3.3.1 An Example 149
3.3.4 Extension to Non-Dominance Alternatives and C Samples 149
3.3.5 Problems and Exercises 151
4 The Nonparametric Combination Methodology 153
4.1 Introduction 153
4.1.1 General Aspects 153
4.1.2 Some Preliminary Notes 155
4.1.3 Main Assumptions and Notation 157
4.1.4 Some Comments 159
4.2 The Nonparametric Combination Methodology 160
4.2.1 Assumptions on Partial Tests 160
4.2.2 Desirable Properties of Combining Functions 162
4.2.3 A Two-Phase Algorithm for Nonparametric Combination 164
4.2.4 Some Useful Combining Functions 168
4.2.5 Why Combination Is Nonparametric 177
4.2.6 On Admissible Combining Functions 178
4.2.7 Problems and Exercises 179
4.3 Consistency, Unbiasedness, and Power of Combined Tests 182
4.3.1 Consistency 182
4.3.2 Unbiasedness 182
4.3.3 A Non-Consistent Combining Function 184
4.3.4 Power of Combined Tests 185
4.3.4.1 The Conditional Power Function 185
4.3.4.2 The Unconditional Power Function 186
4.3.5 Multivariate Conditional Confidence Region for ? 187
4.3.6 Problems and Exercises 188
4.4 Some Further Asymptotic Properties 189
4.4.1 General Conditions 189
4.4.2 Some Asymptotic Properties 190
4.5 Finite-Sample Consistency 194
4.5.1 Introduction and Motivations 194
4.5.2 Finite-Sample Consistency 195
4.5.3 Some Applications of Finite-Sample Consistency 201
4.6 Some Examples of Nonparametric Combination 207
4.6.1 Problems and Exercises 229
4.7 Comments on the Nonparametric Combination 231
4.7.1 General Comments 231
5 Multiple Testing Problems and Multiplicity Adjustment 233
5.1 Defining Raw and Adjusted p-Values 233
5.2 Controlling for Multiplicity 235
5.2.1 Multiple Comparison and Multiple Testing 235
5.2.2 Some Definitions of the Global Type I Error 236
5.3 Multiple Testing 237
5.4 The Closed Testing Approach 238
5.4.1 Closed Testing for Multiple Testing 239
5.4.2 Closed Testing Using the MinP Bonferroni-Holm Procedure 241
5.5 Weighted Methods for Controlling FWE and FDR 245
5.6 Adjusting Stepwise p-Values 247
5.6.1 Showing Biasedness of Standard p-Values for Stepwise Regression 248
5.6.2 Algorithm Description 248
5.6.3 Optimal Subset Procedures 250
6 Analysis of Multivariate Categorical Variables 251
6.1 Introduction 251
6.2 The Multivariate McNemar Test 252
6.2.1 An Extension of the Multivariate McNemar Test 255
6.3 Multivariate Goodness-of-Fit Testing for Ordered Variables 256
6.3.1 Multivariate Extension of Fisher's Exact Probability Test 258
6.4 MANOVA with Nominal Categorical Data 258
6.4.1 A Formal Description 258
6.5 Stochastic Ordering 260
6.5.1 Formal Description 260
6.5.2 Further Breaking Down of the Hypotheses 261
6.5.3 Permutation Test 261
6.6 Multifocus Analysis 264
6.6.1 General Aspects 264
6.6.2 The Multifocus Solution 264
6.6.3 An Application 267
6.7 Isotonic Inference 268
6.7.1 Introduction 268
6.7.2 Allelic Association Analysis in Genetics 270
6.7.3 Parametric Solutions 271
6.7.3.1 Chiano and Clayton's Method 271
6.7.3.2 Maximum-Likelihood Approach 272
6.7.4 Permutation Approach 273
6.8 Test on Moments for Ordered Variables 274
6.8.1 General Aspects 274
6.8.2 Score Transformations and Univariate Tests 275
6.8.3 Multivariate Extension 276
6.9 Heterogeneity Comparisons 278
6.9.1 Introduction 278
6.9.2 Tests for Comparing Heterogeneities 279
6.9.3 A Case Study in Population Genetics 281
6.10 Application to PhD Programme Evaluation 282
6.10.1 Description of the Problem 282
6.10.2 Global Satisfaction Index 284
6.10.3 Multivariate Performance Comparisons 285
7 Permutation Testing for Repeated Measurements 287
7.1 Introduction 287
7.2 Carry-Over Effects in Repeated-Measures Designs 288
7.3 Modelling Repeated Measurements 289
7.3.1 A General Additive Model 289
7.3.2 The Hypotheses of Interest 291
7.4 Testing Solutions 292
7.4.1 Solutions Using the NPC Approach 292
7.4.2 Analysis of Two-Sample Dominance Problems 294
7.4.3 Analysis of the Cross-Over (AB-BA) Design 294
7.4.4 Analysis of a Cross-Over Design with Paired Data 295
7.5 Testing for Repeated Measurements with Missing Data 296
7.5.1 General Aspects of Permutation Testing with Missing Data 297
7.5.2 Bibliographic Notes 297
7.6 On Missing Data Processes 298
7.6.1 Data Missing Completely at Random 299
7.6.2 Data Missing Not at Random 299
7.7 The Permutation Approach 300
7.7.1 Deletion, Imputation, and Intention to Treat Strategies 301
7.7.2 Breaking Down the Hypotheses 303
7.8 The Structure of Testing Problems 303
7.8.1 Hypotheses for MNAR Models 303
7.8.2 Hypotheses for MCAR Models 305
7.8.3 Permutation Structure with Missing Values 305
7.9 Permutation Analysis of Missing Values 307
7.9.1 Partitioning the Permutation Sample Space 307
7.9.2 Solution for Two-Sample MCAR Problems 308
7.9.3 Extensions to Multivariate C-Sample Problems 310
7.9.4 Extension to MNAR Models 311
7.10 'GERMINA' Data: An Example of a MNAR Model 312
7.10.1 Problem Description 313
7.10.2 The Permutation Solution 315
7.11 Multivariate Paired Observations 318
7.11.1 The Problem 318
7.12 Repeated Measures and Missing Data 319
7.12.1 An Example 320
7.13 Botulinum Data: Example 322
7.13.1 Data Description 322
7.13.2 Research Questions and Methods 323
7.14 Waterfalls Data: Example 324
7.14.1 Data Description 324
7.14.2 Research Questions and Methods 325
8 Some Stochastic Ordering Problems 327
8.1 Multivariate Ordered Alternatives 327
8.2 Testing for Umbrella Alternatives 329
8.2.1 Hypotheses and Tests in Simple Stochastic Ordering 331
8.2.2 Permutation Tests for Umbrella Alternatives 332
8.3 Tumour Growth Curves: Example 335
8.4 PERC Data: Example 336
8.4.1 Introduction 336
8.4.2 A Permutation Solution 339
9 NPC Tests for Survival Analysis 341
9.1 Introduction and Main Notation 341
9.1.1 Failure Time Distributions 341
9.1.2 Data Structure 342
9.2 Comparison of Survival Curves 343
9.3 An Overview from the Literature 345
9.3.1 Permutation Tests in Survival Analysis 347
9.4 Two NPC Tests 348
9.4.1 Breaking Down the Hypotheses 349
9.4.2 The Test Structure 350
9.4.3 NPC Test for Treatment-Independent Censoring 350
9.4.4 NPC Test for Treatment-Dependent Censoring 352
9.5 Case Study: In-Hospital Mortality 355
9.5.1 Results 355
9.6 Case Study: Esophageal Cancer 356
9.6.1 A Permutation Solution 358
9.6.2 Survival Analysis with Stratification by Propensity Score 359
9.6.3 Integrating Propensity Score and NPC Testing 361
9.7 Case Study: Comparison of Three Survival Curves 366
9.7.1 Unstratified Survival Analysis 367
9.7.2 Stratified Survival Analysis Using Propensity Score 368
10 NPC Tests in Shape Analysis 373
10.1 A Brief Overview of Statistical Shape Analysis 375
10.1.1 How to Describe Shapes 375
10.1.2 Multivariate Morphometrics 377
10.2 Inference with Shape Data 380
10.3 NPC Approach to Shape Analysis 382
10.3.1 Comparative Simulation Study 384
10.4 NPC Analysis with Correlated Landmarks 387
10.5 An Application to Mediterranean Monk Seal Skulls 390
10.5.1 Some Remarks 393
11 Case Studies 395
11.1 Autofluorescence Data 395
11.1.1 A Permutation Solution 397
11.1.2 Analysis Using R 399
11.2 Confocal Data 402
11.2.1 A Permutation Solution 404
11.2.1.1 Evaluation of Treatment Effect Using R 406
11.2.1.2 Evaluation of Time Effect Using R 409
11.2.2 Two-Way (M)ANOVA 411
11.2.2.1 Permutation Tests in Two-Way (M)ANOVA 411
11.2.2.2 Analysis Using R 413
11.3 SETIG Data 414
11.3.1 Analysis Using R 417
11.4 Logistic Regression and NPC Test for Multivariate Analysis 420
11.4.1 Lymph Node Metastases Data 420
11.4.2 Bladder Cancer Data 423
11.4.2.1 Applying NPC: Some Results 425
11.4.2.2 Analysis with Logistic Regression 425
11.4.2.3 Final Comments 427
12 Permutation Tests Over the Years 431
12.1 Kind of Data 432
12.1.1 Time-Series Data 432
12.1.2 Thick Data 432
12.1.3 Imaging Data 435
12.1.4 Survival Data 436
12.1.5 Censored Data 436
12.2 Different Models 437
12.2.1 Regression 437
12.2.2 Classification 440
12.2.3 Doe 440
12.2.4 Ranking 440
12.3 Different Tests 441
12.3.1 Test for Centrality 441
12.3.2 Test for Variability 446
12.3.3 Test for Both Centrality and Variability 446
12.3.4 ANOVA Test 447
12.3.5 Testing Relationship 447
12.3.6 NPC Tests 448
12.4 Methodological and Computational Aspects 449
12.4.1 Power 449
12.4.2 Computational Problems 452
12.4.3 Other Problems 452
Appendix A Examples with Applications in R 453
A. 1 Mult Data Example 453
A. 2 Washing Test Data 456
A. 3 Germina Data Example 460
A. 4 Botulinum Data 464
A. 5 Waterfalls Data 467
A. 6 Tumour Growth Curves 469
A. 7 PERC Data 471
A. 8 Mediterranean Monk Seal Skulls 475
Bibliography 479
Index 517