Data Profiling
Published on 30. November 2018
Book
Hardback
154 pages
978-1-68173-448-4 (ISBN)
Description
Data profiling refers to the activity of collecting data about data, i.e., metadata. Most IT professionals and researchers who work with data have engaged in data profiling, at least informally, to understand and explore an unfamiliar dataset or to determine whether a new dataset is appropriate for a particular task at hand. Data profiling results are also important in a variety of other situations, including query optimization, data integration, and data cleaning. Simple metadata are statistics, such as the number of rows and columns, schema and datatype information, the number of distinct values, statistical value distributions, and the number of null or empty values in each column. More complex types of metadata are statements about multiple columns and their correlation, such as candidate keys, functional dependencies, and other types of dependencies.
This book provides a classification of the various types of profilable metadata, discusses popular data profiling tasks, and surveys state-of-the-art profiling algorithms. While most of the book focuses on tasks and algorithms for relational data profiling, we also briefly discuss systems and techniques for profiling non-relational data such as graphs and text. We conclude with a discussion of data profiling challenges and directions for future work in this area.
This book provides a classification of the various types of profilable metadata, discusses popular data profiling tasks, and surveys state-of-the-art profiling algorithms. While most of the book focuses on tasks and algorithms for relational data profiling, we also briefly discuss systems and techniques for profiling non-relational data such as graphs and text. We conclude with a discussion of data profiling challenges and directions for future work in this area.
More details
Series
Language
English
Place of publication
San Rafael
United States
Dimensions
Height: 235 mm
Width: 190 mm
Weight
333 gr
ISBN-13
978-1-68173-448-4 (9781681734484)
Copyright in bibliographic data and cover images is held by Nielsen Book Services Limited or by the publishers or by their respective licensors: all rights reserved.
Schweitzer Classification
Persons
Ziawasch Abedjan is Juniorprofessor (Assistant Professor) and Head of the ""Big Data Management"" (BigDaMa) Group at the Technische Universität Berlin. Before Ziawasch was a postdoc at the ""Computer Science and Artificial Intelligence Laboratory"" at MIT working on various data integration topics. Ziawasch received his Ph.D. from the Hasso Plattner Institute in Potsdam, Germany. His research interests include, data mining, data integration, and data profiling.
Lukasz Golab is an Associate Professor at the University of Waterloo and a Canada Research Chair. Prior to joining Waterloo, he was a Senior Member of Research Staff at AT&T Labs in Florham Park, NJ, USA. He holds a B.Sc. in Computer Science (with High Distinction) from the University of Toronto and a Ph.D. in Computer Science (with Alumni Gold Medal) from the University of Waterloo. His publications span several research areas within data management and data analytics, including data stream management, data profiling, data quality, data science for social good, and educational data mining.
Felix Naumann studied mathematics, economy, and computer sciences at the University of Technology in Berlin. After receiving his diploma in 1997 he joined the graduate school ""Distributed Information Systems"" at Humboldt University of Berlin. He completed his Ph.D. thesis on ""Quality-driven Query Answering"" in 2000. In 2001 and 2002 he worked at the IBM Almaden Research Center on topics around data integration. From 2003-2006 he was an assistant professor of information integration at the Humboldt University of Berlin. Since 2006 he has held the chair for information systems at the Hasso Plattner Institute at the University of Potsdam in Germany. He is Editor-in-Chief of the Information Systems journal. His research interests are in the areas of information integration, data quality, data cleansing, text extraction, and-of course-data profiling. He has given numerous invited talks and tutorials on the topic of the book.
Thorsten Papenbrock is a researcher and lecturer at the Hasso Plattner Institute at the University of Potsdam in Germany. He received his M.Sc. in IT-Systems Engineering in 2014 and his Ph.D. in Computer Science in 2017. His thesis on ""Data Profiling-Efficient Discovery of Dependencies"" inspired many sections of this book. In research, his main interests are data profiling, data cleaning, distributed and parallel computing, database systems, and data analytics.
Lukasz Golab is an Associate Professor at the University of Waterloo and a Canada Research Chair. Prior to joining Waterloo, he was a Senior Member of Research Staff at AT&T Labs in Florham Park, NJ, USA. He holds a B.Sc. in Computer Science (with High Distinction) from the University of Toronto and a Ph.D. in Computer Science (with Alumni Gold Medal) from the University of Waterloo. His publications span several research areas within data management and data analytics, including data stream management, data profiling, data quality, data science for social good, and educational data mining.
Felix Naumann studied mathematics, economy, and computer sciences at the University of Technology in Berlin. After receiving his diploma in 1997 he joined the graduate school ""Distributed Information Systems"" at Humboldt University of Berlin. He completed his Ph.D. thesis on ""Quality-driven Query Answering"" in 2000. In 2001 and 2002 he worked at the IBM Almaden Research Center on topics around data integration. From 2003-2006 he was an assistant professor of information integration at the Humboldt University of Berlin. Since 2006 he has held the chair for information systems at the Hasso Plattner Institute at the University of Potsdam in Germany. He is Editor-in-Chief of the Information Systems journal. His research interests are in the areas of information integration, data quality, data cleansing, text extraction, and-of course-data profiling. He has given numerous invited talks and tutorials on the topic of the book.
Thorsten Papenbrock is a researcher and lecturer at the Hasso Plattner Institute at the University of Potsdam in Germany. He received his M.Sc. in IT-Systems Engineering in 2014 and his Ph.D. in Computer Science in 2017. His thesis on ""Data Profiling-Efficient Discovery of Dependencies"" inspired many sections of this book. In research, his main interests are data profiling, data cleaning, distributed and parallel computing, database systems, and data analytics.
Content
- Preface
- Acknowledgments
- Discovering Metadata
- Data Profiling Tasks
- Single Column Analysis
- Dependency Discovery
- Relaxed and Other Dependencies
- Use Cases
- Profiling Non-Relational Data
- Data Profiling Tools
- Data Profiling Challenges
- Conclusions
- Bibliography
- Authors' Biographies
- Acknowledgments
- Discovering Metadata
- Data Profiling Tasks
- Single Column Analysis
- Dependency Discovery
- Relaxed and Other Dependencies
- Use Cases
- Profiling Non-Relational Data
- Data Profiling Tools
- Data Profiling Challenges
- Conclusions
- Bibliography
- Authors' Biographies