Data Engineering and Data Science
Concepts and Applications
1st Edition
Published on 19. September 2023
Book
Hardback
464 pages
978-1-119-84187-6 (ISBN)
Description
DATA ENGINEERING and DATA SCIENCE Written and edited by one of the most prolific and well-known experts in the field and his team, this exciting new volume is the "one-stop shop" for the concepts and applications of data science and engineering for data scientists across many industries.
The field of data science is incredibly broad, encompassing everything from cleaning data to deploying predictive models. However, it is rare for any single data scientist to be working across the spectrum day to day. Data scientists usually focus on a few areas and are complemented by a team of other scientists and analysts. Data engineering is also a broad field, but any individual data engineer doesn't need to know the whole spectrum of skills. Data engineering is the aspect of data science that focuses on practical applications of data collection and analysis. For all the work that data scientists do to answer questions using large sets of information, there have to be mechanisms for collecting and validating that information.
In this exciting new volume, the team of editors and contributors sketch the broad outlines of data engineering, then walk through more specific descriptions that illustrate specific data engineering roles. Data-driven discovery is revolutionizing the modeling, prediction, and control of complex systems. This book brings together machine learning, engineering mathematics, and mathematical physics to integrate modeling and control of dynamical systems with modern methods in data science. It highlights many of the recent advances in scientific computing that enable data-driven methods to be applied to a diverse range of complex systems, such as turbulence, the brain, climate, epidemiology, finance, robotics, and autonomy. Whether for the veteran engineer or scientist working in the field or laboratory, or the student or academic, this is a must-have for any library.
The field of data science is incredibly broad, encompassing everything from cleaning data to deploying predictive models. However, it is rare for any single data scientist to be working across the spectrum day to day. Data scientists usually focus on a few areas and are complemented by a team of other scientists and analysts. Data engineering is also a broad field, but any individual data engineer doesn't need to know the whole spectrum of skills. Data engineering is the aspect of data science that focuses on practical applications of data collection and analysis. For all the work that data scientists do to answer questions using large sets of information, there have to be mechanisms for collecting and validating that information.
In this exciting new volume, the team of editors and contributors sketch the broad outlines of data engineering, then walk through more specific descriptions that illustrate specific data engineering roles. Data-driven discovery is revolutionizing the modeling, prediction, and control of complex systems. This book brings together machine learning, engineering mathematics, and mathematical physics to integrate modeling and control of dynamical systems with modern methods in data science. It highlights many of the recent advances in scientific computing that enable data-driven methods to be applied to a diverse range of complex systems, such as turbulence, the brain, climate, epidemiology, finance, robotics, and autonomy. Whether for the veteran engineer or scientist working in the field or laboratory, or the student or academic, this is a must-have for any library.
More details
Language
English
Place of publication
United States
Publishing group
John Wiley & Sons Inc
Target group
Professional and scholarly
Product notice
sewn/stitched
Cloth over boards
Dimensions
Height: 231 mm
Width: 159 mm
Thickness: 25 mm
Weight
794 gr
ISBN-13
978-1-119-84187-6 (9781119841876)
Copyright in bibliographic data and cover images is held by Nielsen Book Services Limited or by the publishers or by their respective licensors: all rights reserved.
Schweitzer Classification
Other editions
Additional editions
Kukatlapalli Pradeep Kumar | Aynur Unal | Vinay Jha Pillai
Data Engineering and Data Science
Concepts and Applications
E-Book
08/2023
1st Edition
Wiley
€200.99
Available for download
Kukatlapalli Pradeep Kumar | Aynur Unal | Vinay Jha Pillai
Data Engineering and Data Science
Concepts and Applications
E-Book
08/2023
1st Edition
Wiley
€190.99
Available for download
Persons
Kukatlapalli Pradeep Kumar, PhD, is an associate professor and the Program Coordinator for Data Science at Christ University, Bangalore, India. He has 13 years of research and academic experience. He has published in many journals and presented numerous conference papers.
Aynur Unal, PhD, educated at Stanford University (class of '73), has taught at Stanford University for almost 40 years and established the Acoustics Institute. Her work on "New Transform Domains for the Onset of Failures" received a prestigious research award.
Vinay Jha Pillai, PhD, is an associate professor in the Department of Electronics and Communication Engineering at CHRIST University, Bangalore, India. He has 12 years of academic experience and holds two patents. He has also completed two funded projects as principal investigator.
Hari Murthy, PhD, is a faculty member in the Department of Electronics and Communication Engineering, CHRIST University, Bengaluru, India. He finished his PhD from the University of Canterbury, New Zealand where his thesis was on novel anticorrosion materials. He has authored book chapters and published papers in international journals and conferences and has served as part of the program committees for several international conferences.
M. Niranjanamurthy, PhD, is an assistant professor in the Department of Computer Applications, M S Ramaiah Institute of Technology, Bangalore, Karnataka. He earned his PhD in computer science at JJTU, Rajasthan, India. He has over 11 years of teaching experience and two years of industry experience as a software engineer. He has published several books, and he is working on numerous books for Scrivener Publishing. He has published over 60 papers for scholarly journals and conferences, and he is working as a reviewer in 22 scientific journals. He also has numerous awards to his credit.
Aynur Unal, PhD, educated at Stanford University (class of '73), has taught at Stanford University for almost 40 years and established the Acoustics Institute. Her work on "New Transform Domains for the Onset of Failures" received a prestigious research award.
Vinay Jha Pillai, PhD, is an associate professor in the Department of Electronics and Communication Engineering at CHRIST University, Bangalore, India. He has 12 years of academic experience and holds two patents. He has also completed two funded projects as principal investigator.
Hari Murthy, PhD, is a faculty member in the Department of Electronics and Communication Engineering, CHRIST University, Bengaluru, India. He finished his PhD from the University of Canterbury, New Zealand where his thesis was on novel anticorrosion materials. He has authored book chapters and published papers in international journals and conferences and has served as part of the program committees for several international conferences.
M. Niranjanamurthy, PhD, is an assistant professor in the Department of Computer Applications, M S Ramaiah Institute of Technology, Bangalore, Karnataka. He earned his PhD in computer science at JJTU, Rajasthan, India. He has over 11 years of teaching experience and two years of industry experience as a software engineer. He has published several books, and he is working on numerous books for Scrivener Publishing. He has published over 60 papers for scholarly journals and conferences, and he is working as a reviewer in 22 scientific journals. He also has numerous awards to his credit.
Content
Preface xv
1 Quality Assurance in Data Science: Need, Challenges and Focus 1
Jasmine K.S., Ajay D. K. and Aditya Raj
1.1 Introduction 1
1.2 Testing and Quality Assurance 3
1.3 Product Quality and Test Efforts 4
1.4 Data Masking in Data Model and Associated Risks 8
1.5 Prediction in Data Science 9
1.6 Role of Metrics in Evaluation 20
1.7 Quantity of Data in Quality Assurance 20
1.8 Identifying the Right Data Sources 20
1.9 Conclusion 21
2 Design and Implementation of Social Media Mining -- Knowledge Discovery Methods for Effective Digital Marketing Strategies 23
Prashant Bhat and Pradnya Malaganve
2.1 Introduction 24
2.2 Literature Review 26
2.3 Novel Framework for Social Media Data Mining and Knowledge Discovery 29
2.4 Classification for Comparison Analysis 34
2.5 Clustering Methodology to Provide Digital Marketing Strategies 38
2.6 Experimental Results 43
2.7 Conclusion 45
3 A Study on Big Data Engineering Using Cloud Data Warehouse 49
Manjunath T. N., Pushpa S. K., Ravindra S. Hegadi and Ananya Hathwar K. S.
3.1 Introduction 50
3.2 Comparison Study of Different Cloud Data Warehouses 51
3.3 Snowflake Cloud Data Warehouse 55
3.4 Google BigQuery Cloud Data Warehouse 58
3.5 Microsoft Azure Synapse Cloud Data Warehouse 61
3.6 Informatica Intelligent Cloud Services (IICS) 64
3.7 Conclusion 67
4 Data Mining with Cluster Analysis Through Partitioning Approach of Huge Transaction Data 71
Sampath Kini K. and Karthik Pai B.H.
4.1 Introduction 72
4.2 Methodology Used in Proposed Cluster Analysis System 75
4.3 Literature Survey on Existing Systems 80
4.4 Conclusion 82
5 Application of Data Science in Macromodeling of Nonlinear Dynamical Systems 85
Nagaraj S., Seshachalam D. and Jayalatha G.
5.1 Introduction 86
5.2 Nonlinear Autonomous Dynamical System 89
5.3 Nonlinear System - MOR 90
5.4 Data Science Life Cycle 92
5.5 Artificial Neural Network in Modeling 94
5.6 Neuron Spiking Model Using FitzHugh-Nagumo (F-N) System 99
5.7 Ring Oscillator Model 104
5.8 Nonlinear VLSI Interconnect Model Using Telegraph Equation 108
5.9 Macromodel Using Machine Learning 112
5.10 MOR of Dynamical Systems Using POD-ANN 115
5.11 Numerical Results 117
5.12 Conclusion 126
6 Comparative Analysis of Various Ensemble Approaches for Web Page Classification 137
J. Dutta, Yong Woon Kim and Dalia Dominic
6.1 Introduction 138
6.2 Literature Survey 139
6.3 Material and Methods 144
6.4 Ensemble Classifiers 146
6.5 Results 148
6.6 Conclusion 169
7 Feature Engineering and Selection Approach Over Malicious Image 173
P.M. Kavitha and B. Muruganantham
7.1 Introduction 173
7.2 Feature Engineering Techniques 176
7.3 Malicious Feature Engineering 182
7.4 Image Processing Technique 183
7.5 Image Processing Techniques for Analysis on Malicious Images 185
7.6 Conclusion 191
8 Cubic-Regression and Likelihood Based Boosting GAM to Model Drug Sensitivity for Glioblastoma 195
Satyawant Kumar, Vinai George Biju, Ho-Kyoung Lee and Blessy Baby Mathew
8.1 Introduction 196
8.2 Literature Survey 198
8.3 Materials and Methods 201
8.4 Evaluations, Results and Discussions 209
9 Unobtrusive Engagement Detection through Semantic Pose Estimation and Lightweight ResNet for an Online Class Environment 225
Michael Moses Thiruthuvanathan, Balachandran Krishnan and Madhavi Rangaswamy
9.1 Introduction 226
9.2 Related Work 230
9.3 Proposed Methodology 234
9.4 Experimentation 241
9.5 Results and Discussions 245
10 Building Rule Base for Decision Making -- A Fuzzy-Rough Approach 255
Sabu M. K., Neeraj Krishna M. S. and Reshmi R.
10.1 Introduction 256
10.2 Literature Review 258
10.3 Discretization of the Dataset Using Fuzzy Set Theory 260
10.4 Description of the Dataset 260
10.5 Process Involved in Proposed Work 261
10.6 Experiment 262
10.7 Evaluation Result 267
10.8 Discussion 273
11 An Effective Machine Learning Approach to Model Healthcare Data 279
Shaila H. Koppad, S. Anupama Kumar and Mohan Kumar
11.1 Introduction 280
11.2 Types of Data in Healthcare 281
11.3 Big Data in Healthcare 283
11.4 Different V's of Big Data 284
11.5 About COPD 285
11.6 Methodology Implemented 290
12 Recommendation Engine for Retail Domain Using Machine Learning Techniques 303
Chandrashekhara K. T., Gireesh Babu C. N. and Thungamani M.
12.1 Introduction 304
12.2 Proposed System 304
12.3 Results 312
12.3.1 ARIMA Forecasting 312
12.4 Conclusion 313
13 Mining Heterogeneous Lung Cancer from Computer Tomography (CT) Scan with the Confusion Matrix 317
Denny Dominic and Krishnan Balachandran
13.1 Introduction 317
13.2 Literature Review 319
13.3 Methodology 320
13.4 Result 326
13.5 Conclusion and Future Scope 332
References 332
14 ML Algorithms and Their Approach on COVID-19 Data Analysis 335
Kambaluru Ashok, Penumalli Anvesh Reddy and Kukatlapalli Pradeep Kumar
14.1 Introduction 336
14.2 DataSet 336
14.3 Types of Machine Learning Algorithms 338
14.4 Conclusion 348
15 Analysis and Design for the Early Stage Detection of Lung Diseases Using Machine Learning Algorithms 351
Sindhu Madhuri, Mahesh T. R., Vivek V., Shashikala H. K. and C. Saravanan
15.1 Introduction 352
15.2 Machine Learning Algorithms 358
15.3 Evaluation Metrics and Comparative Results for Early Detection of Lung Diseases 364
15.4 Conclusion 369
16 Estimation of Cancer Risk through Artificial Neural Network 373
K. Aditya Shastry, Sanjay H. A., Balaji N. and Karthik Pai B. H.
16.1 Introduction 373
16.2 Case Studies Related to Cancer Risk Estimation Using ANN 375
16.3 Datasets Used in Cancer Risk Estimation 388
16.4 Discussion 397
16.5 Future Scope 400
16.6 Conclusion 400
17 Applications and Advancements in Data Science and Analytics 409
T. Mamatha, A. Balaram, B. Rama Subba Reddy, C. Shoba Bindu and M. Niranjanamurthy
17.1 Data Science and Analytics in Software Testing 410
17.2 Applications of Data Science and Analytics 411
17.3 Selenium Testing Tool in Data Science 419
17.4 Challenges and Advancements in Data Science 425
17.5 Data Science and Analytics Tools 430
17.6 Conclusion 438
References 439
About the Editors 441
Index 443
1 Quality Assurance in Data Science: Need, Challenges and Focus 1
Jasmine K.S., Ajay D. K. and Aditya Raj
1.1 Introduction 1
1.2 Testing and Quality Assurance 3
1.3 Product Quality and Test Efforts 4
1.4 Data Masking in Data Model and Associated Risks 8
1.5 Prediction in Data Science 9
1.6 Role of Metrics in Evaluation 20
1.7 Quantity of Data in Quality Assurance 20
1.8 Identifying the Right Data Sources 20
1.9 Conclusion 21
2 Design and Implementation of Social Media Mining -- Knowledge Discovery Methods for Effective Digital Marketing Strategies 23
Prashant Bhat and Pradnya Malaganve
2.1 Introduction 24
2.2 Literature Review 26
2.3 Novel Framework for Social Media Data Mining and Knowledge Discovery 29
2.4 Classification for Comparison Analysis 34
2.5 Clustering Methodology to Provide Digital Marketing Strategies 38
2.6 Experimental Results 43
2.7 Conclusion 45
3 A Study on Big Data Engineering Using Cloud Data Warehouse 49
Manjunath T. N., Pushpa S. K., Ravindra S. Hegadi and Ananya Hathwar K. S.
3.1 Introduction 50
3.2 Comparison Study of Different Cloud Data Warehouses 51
3.3 Snowflake Cloud Data Warehouse 55
3.4 Google BigQuery Cloud Data Warehouse 58
3.5 Microsoft Azure Synapse Cloud Data Warehouse 61
3.6 Informatica Intelligent Cloud Services (IICS) 64
3.7 Conclusion 67
4 Data Mining with Cluster Analysis Through Partitioning Approach of Huge Transaction Data 71
Sampath Kini K. and Karthik Pai B.H.
4.1 Introduction 72
4.2 Methodology Used in Proposed Cluster Analysis System 75
4.3 Literature Survey on Existing Systems 80
4.4 Conclusion 82
5 Application of Data Science in Macromodeling of Nonlinear Dynamical Systems 85
Nagaraj S., Seshachalam D. and Jayalatha G.
5.1 Introduction 86
5.2 Nonlinear Autonomous Dynamical System 89
5.3 Nonlinear System - MOR 90
5.4 Data Science Life Cycle 92
5.5 Artificial Neural Network in Modeling 94
5.6 Neuron Spiking Model Using FitzHugh-Nagumo (F-N) System 99
5.7 Ring Oscillator Model 104
5.8 Nonlinear VLSI Interconnect Model Using Telegraph Equation 108
5.9 Macromodel Using Machine Learning 112
5.10 MOR of Dynamical Systems Using POD-ANN 115
5.11 Numerical Results 117
5.12 Conclusion 126
6 Comparative Analysis of Various Ensemble Approaches for Web Page Classification 137
J. Dutta, Yong Woon Kim and Dalia Dominic
6.1 Introduction 138
6.2 Literature Survey 139
6.3 Material and Methods 144
6.4 Ensemble Classifiers 146
6.5 Results 148
6.6 Conclusion 169
7 Feature Engineering and Selection Approach Over Malicious Image 173
P.M. Kavitha and B. Muruganantham
7.1 Introduction 173
7.2 Feature Engineering Techniques 176
7.3 Malicious Feature Engineering 182
7.4 Image Processing Technique 183
7.5 Image Processing Techniques for Analysis on Malicious Images 185
7.6 Conclusion 191
8 Cubic-Regression and Likelihood Based Boosting GAM to Model Drug Sensitivity for Glioblastoma 195
Satyawant Kumar, Vinai George Biju, Ho-Kyoung Lee and Blessy Baby Mathew
8.1 Introduction 196
8.2 Literature Survey 198
8.3 Materials and Methods 201
8.4 Evaluations, Results and Discussions 209
9 Unobtrusive Engagement Detection through Semantic Pose Estimation and Lightweight ResNet for an Online Class Environment 225
Michael Moses Thiruthuvanathan, Balachandran Krishnan and Madhavi Rangaswamy
9.1 Introduction 226
9.2 Related Work 230
9.3 Proposed Methodology 234
9.4 Experimentation 241
9.5 Results and Discussions 245
10 Building Rule Base for Decision Making -- A Fuzzy-Rough Approach 255
Sabu M. K., Neeraj Krishna M. S. and Reshmi R.
10.1 Introduction 256
10.2 Literature Review 258
10.3 Discretization of the Dataset Using Fuzzy Set Theory 260
10.4 Description of the Dataset 260
10.5 Process Involved in Proposed Work 261
10.6 Experiment 262
10.7 Evaluation Result 267
10.8 Discussion 273
11 An Effective Machine Learning Approach to Model Healthcare Data 279
Shaila H. Koppad, S. Anupama Kumar and Mohan Kumar
11.1 Introduction 280
11.2 Types of Data in Healthcare 281
11.3 Big Data in Healthcare 283
11.4 Different V's of Big Data 284
11.5 About COPD 285
11.6 Methodology Implemented 290
12 Recommendation Engine for Retail Domain Using Machine Learning Techniques 303
Chandrashekhara K. T., Gireesh Babu C. N. and Thungamani M.
12.1 Introduction 304
12.2 Proposed System 304
12.3 Results 312
12.3.1 ARIMA Forecasting 312
12.4 Conclusion 313
13 Mining Heterogeneous Lung Cancer from Computer Tomography (CT) Scan with the Confusion Matrix 317
Denny Dominic and Krishnan Balachandran
13.1 Introduction 317
13.2 Literature Review 319
13.3 Methodology 320
13.4 Result 326
13.5 Conclusion and Future Scope 332
References 332
14 ML Algorithms and Their Approach on COVID-19 Data Analysis 335
Kambaluru Ashok, Penumalli Anvesh Reddy and Kukatlapalli Pradeep Kumar
14.1 Introduction 336
14.2 DataSet 336
14.3 Types of Machine Learning Algorithms 338
14.4 Conclusion 348
15 Analysis and Design for the Early Stage Detection of Lung Diseases Using Machine Learning Algorithms 351
Sindhu Madhuri, Mahesh T. R., Vivek V., Shashikala H. K. and C. Saravanan
15.1 Introduction 352
15.2 Machine Learning Algorithms 358
15.3 Evaluation Metrics and Comparative Results for Early Detection of Lung Diseases 364
15.4 Conclusion 369
16 Estimation of Cancer Risk through Artificial Neural Network 373
K. Aditya Shastry, Sanjay H. A., Balaji N. and Karthik Pai B. H.
16.1 Introduction 373
16.2 Case Studies Related to Cancer Risk Estimation Using ANN 375
16.3 Datasets Used in Cancer Risk Estimation 388
16.4 Discussion 397
16.5 Future Scope 400
16.6 Conclusion 400
17 Applications and Advancements in Data Science and Analytics 409
T. Mamatha, A. Balaram, B. Rama Subba Reddy, C. Shoba Bindu and M. Niranjanamurthy
17.1 Data Science and Analytics in Software Testing 410
17.2 Applications of Data Science and Analytics 411
17.3 Selenium Testing Tool in Data Science 419
17.4 Challenges and Advancements in Data Science 425
17.5 Data Science and Analytics Tools 430
17.6 Conclusion 438
References 439
About the Editors 441
Index 443