Competing with High Quality Data
Concepts, Tools, and Techniques for Building a Successful Approach to Data Quality
1st Edition
Published on 15. April 2014
Book
Hardback
304 pages
978-1-118-34232-9 (ISBN)
Description
Create a competitive advantage with data quality Data is rapidly becoming the powerhouse of industry, but low-quality data can actually put a company at a disadvantage. To be used effectively, data must accurately reflect the real-world scenario it represents, and it must be in a form that is usable and accessible. Quality data involves asking the right questions, targeting the correct parameters, and having an effective internal management, organization, and access system. It must be relevant, complete, and correct, while falling in line with pervasive regulatory oversight programs.
Competing with High Quality Data: Concepts, Tools and Techniques for Building a Successful Approach to Data Quality takes a holistic approach to improving data quality, from collection to usage. Author Rajesh Jugulum is globally-recognized as a major voice in the data quality arena, with high-level backgrounds in international corporate finance. In the book, Jugulum provides a roadmap to data quality innovation, covering topics such as:
The four-phase approach to data quality control
Methodology that produces data sets for different aspects of a business
Streamlined data quality assessment and issue resolution
A structured, systematic, disciplined approach to effective data gathering
The book also contains real-world case studies to illustrate how companies across a broad range of sectors have employed data quality systems, whether or not they succeeded, and what lessons were learned. High-quality data increases value throughout the information supply chain, and the benefits extend to the client, employee, and shareholder. Competing with High Quality Data: Concepts, Tools and Techniques for Building a Successful Approach to Data Quality provides the information and guidance necessary to formulate and activate an effective data quality plan today.
Competing with High Quality Data: Concepts, Tools and Techniques for Building a Successful Approach to Data Quality takes a holistic approach to improving data quality, from collection to usage. Author Rajesh Jugulum is globally-recognized as a major voice in the data quality arena, with high-level backgrounds in international corporate finance. In the book, Jugulum provides a roadmap to data quality innovation, covering topics such as:
The four-phase approach to data quality control
Methodology that produces data sets for different aspects of a business
Streamlined data quality assessment and issue resolution
A structured, systematic, disciplined approach to effective data gathering
The book also contains real-world case studies to illustrate how companies across a broad range of sectors have employed data quality systems, whether or not they succeeded, and what lessons were learned. High-quality data increases value throughout the information supply chain, and the benefits extend to the client, employee, and shareholder. Competing with High Quality Data: Concepts, Tools and Techniques for Building a Successful Approach to Data Quality provides the information and guidance necessary to formulate and activate an effective data quality plan today.
More details
Language
English
Place of publication
New York
United States
Target group
Professional and scholarly
Product notice
Unsewn / adhesive bound
Paper over boards
Dimensions
Height: 241 mm
Width: 159 mm
Thickness: 25 mm
Weight
562 gr
ISBN-13
978-1-118-34232-9 (9781118342329)
Copyright in bibliographic data and cover images is held by Nielsen Book Services Limited or by the publishers or by their respective licensors: all rights reserved.
Schweitzer Classification
Other editions
Additional editions
Rajesh Jugulum
Competing with High Quality Data
Concepts, Tools, and Techniques for Building a Successful Approach to Data Quality
E-Book
03/2014
Wiley
€86.99
Available for download
Rajesh Jugulum
Competing with High Quality Data
Concepts, Tools, and Techniques for Building a Successful Approach to Data Quality
E-Book
02/2014
Wiley
€86.99
Available for download
Person
DR. RAJESH JUGULUM, Ph.D., is a Data Quality and Analytics Professional. Rajesh held executive positions in these fields at Citi Group and Bank of America. Before joining financial industry, Rajesh was with MIT where he was involved in research and teaching. Currently, he teaches at Northeastern University in Boston. His honors include 2002 American Society for Quality's Feigenbaum medal and 2006 International Technology Institute's Rockwell medal.
Content
Foreword xiii Prelude xv
Preface xvii
Acknowledgments xix
1 The Importance of Data Quality 1
1.0 Introduction 1
1.1 Understanding the Implications of Data Quality 1
1.2 The Data Management Function 4
1.3 The Solution Strategy 6
1.4 Guide to This Book 6
Section I Building a Data Quality Program 2 The Data Quality Operating Model 13
2.0 Introduction 13
2.1 Data Quality Foundational Capabilities 13
2.1.1 Program Strategy and Governance 14
2.1.2 Skilled Data Quality Resources 14
2.1.3 Technology Infrastructure and Metadata 15
2.1.4 Data Profi ling and Analytics 15
2.1.5 Data Integration 15
2.1.6 Data Assessment 16
2.1.7 Issues Resolution (IR) 16
2.1.8 Data Quality Monitoring and Control 16
2.2 The Data Quality Methodology 17
2.2.1 Establish a Data Quality Program 17
2.2.2 Conduct a Current-State Analysis 17
2.2.3 Strengthen Data Quality Capability through Data Quality Projects 18
2.2.4 Monitor the Ongoing Production Environment and Measure Data Quality Improvement Effectiveness 18
2.2.5 Detailed Discussion on Establishing the Data Quality Program 18
2.2.6 Assess the Current State of Data Quality 21
2.3 Conclusions 22
3 The DAIC Approach 23
3.0 Introduction 23
3.1 Six Sigma Methodologies 23
3.1.1 Development of Six Sigma Methodologies 25
3.2 DAIC Approach for Data Quality 28
3.2.1 The Defi ne Phase 28
3.2.2 The Assess Phase 31
3.2.3 The Improve Phase 36
3.2.4 The Control Phase (Monitor and Measure) 37
3.3 Conclusions 40
Section II Executing a Data Quality Program 4 Quantification of the Impact of Data Quality 43
4.0 Introduction 43
4.1 Building a Data Quality Cost Quantifi cation Framework 43
4.1.1 The Cost Waterfall 44
4.1.2 Prioritization Matrix 46
4.1.3 Remediation and Return on Investment 50
4.2 A Trading Offi ce Illustrative Example 51
4.3 Conclusions 54
5 Statistical Process Control and Its Relevance in Data Quality Monitoring and Reporting 55
5.0 Introduction 55
5.1 What Is Statistical Process Control? 55
5.1.1 Common Causes and Special Causes 57
5.2 Control Charts 59
5.2.1 Different Types of Data 59
5.2.2 Sample and Sample Parameters 60
5.2.3 Construction of Attribute Control Charts 62
5.2.4 Construction of Variable Control Charts 65
5.2.5 Other Control Charts 67
5.2.6 Multivariate Process Control Charts 69
5.3 Relevance of Statistical Process Control in Data Quality Monitoring and Reporting 69
5.4 Conclusions 70
6 Critical Data Elements: Identification, Validation, and Assessment 71
6.0 Introduction 71
6.1 Identifi cation of Critical Data Elements 71
6.1.1 Data Elements and Critical Data Elements 71
6.1.2 CDE Rationalization Matrix 72
6.2 Assessment of Critical Data Elements 75
6.2.1 Data Quality Dimensions 76
6.2.2 Data Quality Business Rules 78
6.2.3 Data Profi ling 79
6.2.4 Measurement of Data Quality Scores 80
6.2.5 Results Recording and Reporting (Scorecard) 80
6.3 Conclusions 82
7 Prioritization of Critical Data Elements (Funnel Approach) 83
7.0 Introduction 83
7.1 The Funnel Methodology (Statistical Analysis for CDE Reduction) 83
7.1.1 Correlation and Regression Analysis for Continuous CDEs 85
7.1.2 Association Analysis for Discrete CDEs 88
7.1.3 Signal-to-Noise Ratios Analysis 90
7.2 Case Study: Basel II 91
7.2.1 Basel II: CDE Rationalization Matrix 91
7.2.2 Basel II: Correlation and Regression Analysis 94
7.2.3 Basel II: Signal-to-Noise (S/N) Ratios 96
7.3 Conclusions 99
8 Data Quality Monitoring and Reporting Scorecards 101
8.0 Introduction 101
8.1 Development of the DQ Scorecards 102
8.2 Analytical Framework (ANOVA, SPCs, Thresholds, Heat Maps) 102
8.2.1 Thresholds and Heat Maps 103
8.2.2 Analysis of Variance (ANOVA) and SPC Charts 107
8.3 Application of the Framework 109
8.4 Conclusions 112
9 Data Quality Issue Resolution 113
9.0 Introduction 113
9.1 Description of the Methodology 113
9.2 Data Quality Methodology 114
9.3 Process Quality/Six Sigma Approach 115
9.4 Case Study: Issue Resolution Process Reengineering 117
9.5 Conclusions 119
10 Information System Testing 121
10.0 Introduction 121
10.1 Typical System Arrangement 122
10.1.1 The Role of Orthogonal Arrays 123
10.2 Method of System Testing 123
10.2.1 Study of Two-Factor Combinations 123
10.2.2 Construction of Combination Tables 124
10.3 MTS Software Testing 126
10.4 Case Study: A Japanese Software Company 130
10.5 Case Study: A Finance Company 133
10.6 Conclusions 138
11 Statistical Approach for Data Tracing 139
11.0 Introduction 139
11.1 Data Tracing Methodology 139
11.1.1 Statistical Sampling 142
11.2 Case Study: Tracing 144
11.2.1 Analysis of Test Cases and CDE Prioritization 144
11.3 Data Lineage through Data Tracing 149
11.4 Conclusions 151
12 Design and Development of Multivariate Diagnostic Systems 153
12.0 Introduction 153
12.1 The Mahalanobis-Taguchi Strategy 153
12.1.1 The Gram Schmidt Orthogonalization Process 155
12.2 Stages in MTS 158
12.3 The Role of Orthogonal Arrays and Signal-to-Noise Ratio in Multivariate Diagnosis 159
12.3.1 The Role of Orthogonal Arrays 159
12.3.2 The Role of S/N Ratios in MTS 161
12.3.3 Types of S/N Ratios 162
12.3.4 Direction of Abnormals 164
12.4 A Medical Diagnosis Example 172
12.5 Case Study: Improving Client Experience 175
12.5.1 Improvements Made Based on Recommendations from MTS Analysis 177
12.6 Case Study: Understanding the Behavior Patterns of Defaulting Customers 178
12.7 Case Study: Marketing 180
12.7.1 Construction of the Reference Group 181
12.7.2 Validation of the Scale 181
12.7.3 Identification of Useful Variables 181
12.8 Case Study: Gear Motor Assembly 182
12.8.1 Apparatus 183
12.8.2 Sensors 184
12.8.3 High-Resolution Encoder 184
12.8.4 Life Test 185
12.8.5 Characterization 185
12.8.6 Construction of the Reference Group or Mahalanobis Space 186
12.8.7 Validation of the MTS Scale 187
12.8.8 Selection of Useful Variables 188
12.9 Conclusions 189
13 Data Analytics 191
13.0 Introduction 191
13.1 Data and Analytics as Key Resources 191
13.1.1 Different Types of Analytics 193
13.1.2 Requirements for Executing Analytics 195
13.1.3 Process of Executing Analytics 196
13.2 Data Innovation 197
13.2.1 Big Data 198
13.2.2 Big Data Analytics 199
13.2.3 Big Data Analytics Operating Model 206
13.2.4 Big Data Analytics Projects: Examples 207
13.3 Conclusions 208
14. Building a Data Quality Practices Center 209
14.0 Introduction 209
14.1 Building a DQPC 209
14.2 Conclusions 211
Appendix A 213
Equations for Signal-to-Noise (S/N) Ratios 213
Nondynamic S/N Ratios 213
Dynamic S/N Ratios 214
Appendix B 217
Matrix Theory: Related Topics 217
What Is a Matrix? 217
Appendix C 221
Some Useful Orthogonal Arrays 221
Two-Level Orthogonal Arrays 221
Three-Level Orthogonal Arrays 255
Index of Terms and Symbols 259
References 261
Index 267
Preface xvii
Acknowledgments xix
1 The Importance of Data Quality 1
1.0 Introduction 1
1.1 Understanding the Implications of Data Quality 1
1.2 The Data Management Function 4
1.3 The Solution Strategy 6
1.4 Guide to This Book 6
Section I Building a Data Quality Program 2 The Data Quality Operating Model 13
2.0 Introduction 13
2.1 Data Quality Foundational Capabilities 13
2.1.1 Program Strategy and Governance 14
2.1.2 Skilled Data Quality Resources 14
2.1.3 Technology Infrastructure and Metadata 15
2.1.4 Data Profi ling and Analytics 15
2.1.5 Data Integration 15
2.1.6 Data Assessment 16
2.1.7 Issues Resolution (IR) 16
2.1.8 Data Quality Monitoring and Control 16
2.2 The Data Quality Methodology 17
2.2.1 Establish a Data Quality Program 17
2.2.2 Conduct a Current-State Analysis 17
2.2.3 Strengthen Data Quality Capability through Data Quality Projects 18
2.2.4 Monitor the Ongoing Production Environment and Measure Data Quality Improvement Effectiveness 18
2.2.5 Detailed Discussion on Establishing the Data Quality Program 18
2.2.6 Assess the Current State of Data Quality 21
2.3 Conclusions 22
3 The DAIC Approach 23
3.0 Introduction 23
3.1 Six Sigma Methodologies 23
3.1.1 Development of Six Sigma Methodologies 25
3.2 DAIC Approach for Data Quality 28
3.2.1 The Defi ne Phase 28
3.2.2 The Assess Phase 31
3.2.3 The Improve Phase 36
3.2.4 The Control Phase (Monitor and Measure) 37
3.3 Conclusions 40
Section II Executing a Data Quality Program 4 Quantification of the Impact of Data Quality 43
4.0 Introduction 43
4.1 Building a Data Quality Cost Quantifi cation Framework 43
4.1.1 The Cost Waterfall 44
4.1.2 Prioritization Matrix 46
4.1.3 Remediation and Return on Investment 50
4.2 A Trading Offi ce Illustrative Example 51
4.3 Conclusions 54
5 Statistical Process Control and Its Relevance in Data Quality Monitoring and Reporting 55
5.0 Introduction 55
5.1 What Is Statistical Process Control? 55
5.1.1 Common Causes and Special Causes 57
5.2 Control Charts 59
5.2.1 Different Types of Data 59
5.2.2 Sample and Sample Parameters 60
5.2.3 Construction of Attribute Control Charts 62
5.2.4 Construction of Variable Control Charts 65
5.2.5 Other Control Charts 67
5.2.6 Multivariate Process Control Charts 69
5.3 Relevance of Statistical Process Control in Data Quality Monitoring and Reporting 69
5.4 Conclusions 70
6 Critical Data Elements: Identification, Validation, and Assessment 71
6.0 Introduction 71
6.1 Identifi cation of Critical Data Elements 71
6.1.1 Data Elements and Critical Data Elements 71
6.1.2 CDE Rationalization Matrix 72
6.2 Assessment of Critical Data Elements 75
6.2.1 Data Quality Dimensions 76
6.2.2 Data Quality Business Rules 78
6.2.3 Data Profi ling 79
6.2.4 Measurement of Data Quality Scores 80
6.2.5 Results Recording and Reporting (Scorecard) 80
6.3 Conclusions 82
7 Prioritization of Critical Data Elements (Funnel Approach) 83
7.0 Introduction 83
7.1 The Funnel Methodology (Statistical Analysis for CDE Reduction) 83
7.1.1 Correlation and Regression Analysis for Continuous CDEs 85
7.1.2 Association Analysis for Discrete CDEs 88
7.1.3 Signal-to-Noise Ratios Analysis 90
7.2 Case Study: Basel II 91
7.2.1 Basel II: CDE Rationalization Matrix 91
7.2.2 Basel II: Correlation and Regression Analysis 94
7.2.3 Basel II: Signal-to-Noise (S/N) Ratios 96
7.3 Conclusions 99
8 Data Quality Monitoring and Reporting Scorecards 101
8.0 Introduction 101
8.1 Development of the DQ Scorecards 102
8.2 Analytical Framework (ANOVA, SPCs, Thresholds, Heat Maps) 102
8.2.1 Thresholds and Heat Maps 103
8.2.2 Analysis of Variance (ANOVA) and SPC Charts 107
8.3 Application of the Framework 109
8.4 Conclusions 112
9 Data Quality Issue Resolution 113
9.0 Introduction 113
9.1 Description of the Methodology 113
9.2 Data Quality Methodology 114
9.3 Process Quality/Six Sigma Approach 115
9.4 Case Study: Issue Resolution Process Reengineering 117
9.5 Conclusions 119
10 Information System Testing 121
10.0 Introduction 121
10.1 Typical System Arrangement 122
10.1.1 The Role of Orthogonal Arrays 123
10.2 Method of System Testing 123
10.2.1 Study of Two-Factor Combinations 123
10.2.2 Construction of Combination Tables 124
10.3 MTS Software Testing 126
10.4 Case Study: A Japanese Software Company 130
10.5 Case Study: A Finance Company 133
10.6 Conclusions 138
11 Statistical Approach for Data Tracing 139
11.0 Introduction 139
11.1 Data Tracing Methodology 139
11.1.1 Statistical Sampling 142
11.2 Case Study: Tracing 144
11.2.1 Analysis of Test Cases and CDE Prioritization 144
11.3 Data Lineage through Data Tracing 149
11.4 Conclusions 151
12 Design and Development of Multivariate Diagnostic Systems 153
12.0 Introduction 153
12.1 The Mahalanobis-Taguchi Strategy 153
12.1.1 The Gram Schmidt Orthogonalization Process 155
12.2 Stages in MTS 158
12.3 The Role of Orthogonal Arrays and Signal-to-Noise Ratio in Multivariate Diagnosis 159
12.3.1 The Role of Orthogonal Arrays 159
12.3.2 The Role of S/N Ratios in MTS 161
12.3.3 Types of S/N Ratios 162
12.3.4 Direction of Abnormals 164
12.4 A Medical Diagnosis Example 172
12.5 Case Study: Improving Client Experience 175
12.5.1 Improvements Made Based on Recommendations from MTS Analysis 177
12.6 Case Study: Understanding the Behavior Patterns of Defaulting Customers 178
12.7 Case Study: Marketing 180
12.7.1 Construction of the Reference Group 181
12.7.2 Validation of the Scale 181
12.7.3 Identification of Useful Variables 181
12.8 Case Study: Gear Motor Assembly 182
12.8.1 Apparatus 183
12.8.2 Sensors 184
12.8.3 High-Resolution Encoder 184
12.8.4 Life Test 185
12.8.5 Characterization 185
12.8.6 Construction of the Reference Group or Mahalanobis Space 186
12.8.7 Validation of the MTS Scale 187
12.8.8 Selection of Useful Variables 188
12.9 Conclusions 189
13 Data Analytics 191
13.0 Introduction 191
13.1 Data and Analytics as Key Resources 191
13.1.1 Different Types of Analytics 193
13.1.2 Requirements for Executing Analytics 195
13.1.3 Process of Executing Analytics 196
13.2 Data Innovation 197
13.2.1 Big Data 198
13.2.2 Big Data Analytics 199
13.2.3 Big Data Analytics Operating Model 206
13.2.4 Big Data Analytics Projects: Examples 207
13.3 Conclusions 208
14. Building a Data Quality Practices Center 209
14.0 Introduction 209
14.1 Building a DQPC 209
14.2 Conclusions 211
Appendix A 213
Equations for Signal-to-Noise (S/N) Ratios 213
Nondynamic S/N Ratios 213
Dynamic S/N Ratios 214
Appendix B 217
Matrix Theory: Related Topics 217
What Is a Matrix? 217
Appendix C 221
Some Useful Orthogonal Arrays 221
Two-Level Orthogonal Arrays 221
Three-Level Orthogonal Arrays 255
Index of Terms and Symbols 259
References 261
Index 267