Smart Grid using Big Data Analytics

Preface xv

Acknowledgments xix

Some Notation xxi

1 Introduction 1

1.1 Big Data: Basic Concepts 1

1.2 Data Mining with Big Data 9

1.3 A Mathematical Introduction to Big Data 13

1.4 A Mathematical Theory of Big Data 28

1.5 Smart Grid 34

1.6 Big Data and Smart Grid 36

1.7 Reading Guide 37

Bibliographical Remarks 39

Part I Fundamentals of Big Data 41

2 The Mathematical Foundations of Big Data Systems 43

2.1 Big Data Analytics 44

2.2 Big Data: Sense, Collect, Store, and Analyze 45

2.3 Intelligent Algorithms 48

2.4 Signal Processing for Smart Grid 48

2.5 Monitoring and Optimization for Power Grids 48

2.6 Distributed Sensing and Measurement for Power Grids 49

2.7 Real-time Analysis of Streaming Data 50

2.8 Salient Features of Big Data 51

2.9 Big Data for Quantum Systems 54

2.10 Big Data for Financial Systems 55

2.11 Big Data for Atmospheric Systems 73

2.12 Big Data for Sensing Networks 74

2.13 Big Data forWireless Networks 75

2.14 Big Data for Transportation 78

Bibliographical Remarks 78

3 Large Random Matrices: An Introduction 79

3.1 Modeling of Large Dimensional Data as Random Matrices 79

3.2 A Brief of Random MatrixTheory 81

3.3 Change Point of Views: From Vectors to Measures 85

3.4 The Stieltjes Transform of Measures 86

3.5 A Fundamental Result: The Marchenko-Pastur Equation 88

3.6 Linear Eigenvalue Statistics and Limit Laws 89

3.7 Central LimitTheorem for Linear Eigenvalue Statistics 99

3.8 Central LimitTheorem for Random Matrix S-1T 101

3.9 Independence for Random Matrices 103

3.10 Matrix-Valued Gaussian Distribution 110

3.11 Matrix-ValuedWishart Distribution 112

3.12 Moment Method 112

3.13 Stieltjes Transform Method 113

3.14 Concentration of the Spectral Measure for Large Random Matrices 114

3.15 Future Directions 117

Bibliographical Remarks 117

4 Linear Spectral Statistics of the Sample Covariance Matrix 121

4.1 Linear Spectral Statistics 121

4.2 Generalized Marchenko-Pastur Distributions 122

4.3 Estimation of Spectral Density Functions 127

4.4 Limiting Spectral Distribution of Time Series 146

Bibliographical Remarks 154

5 Large Hermitian Random Matrices and Free Random Variables 155

5.1 Large Economic/Financial Systems 156

5.2 Matrix-Valued Probability 157

5.3 Wishart-Levy Free Stable Random Matrices 166

5.4 Basic Concepts for Free Random Variables 168

5.5 The Analytical Spectrum of theWishart-Levy Random Matrix 172

5.6 Basic Properties of the Stieltjes Transform 176

5.7 Basic Theorems for the Stieltjes Transform 179

5.8 Free Probability for Hermitian Random Matrices 185

5.9 Random Vandermonde Matrix 196

5.10 Non-Asymptotic Analysis of State Estimation 200

Bibliographical Remarks 201

6 Large Non-Hermitian Random Matrices and Quatartenionic Free Probability Theory 203

6.1 Quatartenionic Free ProbabilityTheory 204

6.2 R-diagonalMatrices 209

6.3 The Sum of Non-Hermitian Random Matrices 216

6.4 The Product of Non-Hermitian Random Matrices 220

6.5 Singular Value Equivalent Models 226

6.6 The Power of the Non-Hermitian Random Matrix 234

6.7 Power Series of Large Non-Hermitian Random Matrices 239

6.8 Products of Random Ginibre Matrices 246

6.9 Products of Rectangular Gaussian Random Matrices 249

6.10 Product of ComplexWishart Matrices 252

6.11 Spectral Relations between Products and Powers 254

6.12 Products of Finite-Size I.I.D. Gaussian Random Matrices 258

6.13 Lyapunov Exponents for Products of Complex Gaussian Random Matrices 260

6.14 Euclidean Random Matrices 264

6.15 Random Matrices with Independent Entries and the Circular Law 273

6.16 The Circular Law and Outliers 275

6.17 Random SVD, Single Ring Law, and Outliers 285

6.18 The Elliptic Law and Outliers 295

Bibliographical Remarks 305

7 The Mathematical Foundations of Data Collection 307

7.1 Architectures and Applications for Big Data 307

7.2 Covariance Matrix Estimation 308

7.3 Spectral Estimators for Large Random Matrices 312

7.4 Asymptotic Framework for Matrix Reconstruction 319

7.5 Optimum Shrinkage 329

7.6 A Shrinkage Approach to Large-Scale Covariance Matrix Estimation 331

7.7 Eigenvectors of Large Sample Covariance Matrix Ensembles 338

7.8 A General Class of Random Matrices 351

Bibliographical Remarks 359

8 Matrix Hypothesis Testing using Large RandomMatrices 361

8.1 Motivating Examples 362

8.2 Hypothesis Test of Two Alternative Random Matrices 363

8.3 Eigenvalue Bounds for Expectation and Variance 364

8.4 Concentration of Empirical Distribution Functions 369

8.5 Random Quadratic Forms 381

8.6 Log-Determinant of Random Matrices 382

8.7 General MANOVA Matrices 383

8.8 Finite Rank Perturbations of Large Random Matrices 386

8.9 Hypothesis Tests for High-Dimensional Datasets 391

8.9.1 Motivation for Likelihood Ratio Test (LRT) and Covariance Matrix Tests 392

8.10 Roy's Largest Root Test 428

8.11 Optimal Tests of Hypotheses for Large Random Matrices 431

8.12 Matrix Elliptically Contoured Distributions 444

8.13 Hypothesis Testing for Matrix Elliptically Contoured Distributions 446

Bibliographical Remarks 452

Part II Smart Grid 455

9 Applications and Requirements of Smart Grid 457

9.1 History 457

9.2 Concepts and Vision 458

9.3 Today's Electric Grid 459

9.4 Future Smart Electrical Energy System 464

10 Technical Challenges for Smart Grid 471

Bibliographical Remarks 483

11 Big Data for Smart Grid 485

11.1 Power in Numbers: Big Data and Grid Infrastructure 485

11.2 Energy's Internet:The Convergence of Big Data and the Cloud 486

11.3 Edge Analytics: Consumers, Electric Vehicles, and Distributed Generation 486

11.4 CrosscuttingThemes: Big Data 486

11.5 Cloud Computing for Smart Grid 488

11.6 Data Storage, Data Access and Data Analysis 488

11.7 The State-of-the-Art Processing Techniques of Big Data 488

11.8 Big Data Meets the Smart Electrical Grid 488

11.9 4Vs of Big Data: Volume, Variety, Value and Velocity 489

11.10 Cloud Computing for Big Data 490

11.11 Big Data for Smart Grid 490

11.12 Information Platforms for Smart Grid 491

Bibliographical Remarks 491

12 Grid Monitoring and State Estimation 493

12.1 Phase Measurement Unit 493

12.2 Optimal PMU Placement 495

12.3 State Estimation 495

12.4 Basics of State Estimation 495

12.5 Evolution of State Estimation 496

12.6 Static State Estimation 497

12.7 Forecasting-Aided State Estimation 500

12.8 Phasor Measurement Units 501

12.9 Distributed System State Estimation 502

12.10 Event-Triggered Approaches to State Estimation 502

12.11 Bad Data Detection 502

12.12 Improved Bad Data Detection 504

12.13 Cyber-Attacks 504

12.14 Line Outage Detection 504

Bibliographical Remarks 504

13 False Data Injection Attacks against State Estimation 505

13.1 State Estimation 505

13.2 False Data Injection Attacks 507

13.3 MMSE State Estimation and Generalized Likelihood Ratio Test 508

13.4 Sparse Recovery from Nonlinear Measurements 512

13.5 Real-Time Intrusion Detection 515

Bibliographical Remarks 515

14 Demand Response 517

14.1 Why Engage Demand? 517

14.2 Optimal Real-time Pricing Algorithms 520

14.3 Transportation Electrification and Vehicle-to-Grid Applications 522

14.4 Grid Storage 522

Bibliographical Remarks 523

Part III Communications and Sensing 525

15 Big Data for Communications 527

15.1 5G and Big Data 527

15.2 5GWireless Communication Networks 527

15.3 Massive Multiple Input, Multiple Output 528

15.4 Free Probability for the Capacity of the Massive MIMO Channel 537

15.5 Spectral Sensing for Cognitive Radio 539

Bibliographical Remarks 539

16 Big Data for Sensing 541

16.1 Distributed Detection and Estimation 541

16.2 Euclidean Random Matrix 547

16.3 Decentralized Computing 548

Appendix A: Some Basic Results on Free Probability 551

Appendix B: Matrix-Valued Random Variables 557

References 567

Index 601