Data-Intensive Workflow Management
For Clouds and Data-Intensive and Scalable Computing Environments
Published on 30. May 2019
Book
Paperback/Softback
179 pages
978-1-68173-557-3 (ISBN)
Description
Workflows may be defined as abstractions used to model the coherent flow of activities in the context of an in silico scientific experiment.They are employed in many domains of science such as bioinformatics, astronomy, and engineering. Such workflows usually present a considerable number of activities and activations (i.e., tasks associated with activities) and may need a long time for execution. Due to the continuous need to store and process data efficiently (making them data-intensive workflows), high-performance computing environments allied to parallelization techniques are used to run these workflows. At the beginning of the 2010s, cloud technologies emerged as a promising environment to run scientific workflows. By using clouds, scientists have expanded beyond single parallel computers to hundreds or even thousands of virtual machines.More recently, Data-Intensive Scalable Computing (DISC) frameworks (e.g., Apache Spark and Hadoop) and environments emerged and are being used to execute data-intensive workflows. DISC environments are composed of processors and disks in large-commodity computing clusters connected using high-speed communications switches and networks. The main advantage of DISC frameworks is that they support and grant efficient in-memory data management for large-scale applications, such as data-intensive workflows. However, the execution of workflows in cloud and DISC environments raise many challenges such as scheduling workflow activities and activations, managing produced data, collecting provenance data, etc.Several existing approaches deal with the challenges mentioned earlier. This way, there is a real need for understanding how to manage these workflows and various big data platforms that have been developed and introduced. As such, this book can help researchers understand how linking workflow management with Data-Intensive Scalable Computing can help in understanding and analyzing scientific big data.In this book, we aim to identify and distill the body of work on workflow management in clouds and DISC environments. We start by discussing the basic principles of data-intensive scientific workflows. Next, we present two workflows that are executed in a single site and multi-site clouds taking advantage of provenance. Afterward, we go towards workflow management in DISC environments, and we present, in detail, solutions that enable the optimized execution of the workflow using frameworks such as Apache Spark and its extensions.
More details
Series
Language
English
Place of publication
San Rafael
United States
Target group
Professional and scholarly
Dimensions
Height: 235 mm
Width: 191 mm
Weight
1 gr
ISBN-13
978-1-68173-557-3 (9781681735573)
Copyright in bibliographic data and cover images is held by Nielsen Book Services Limited or by the publishers or by their respective licensors: all rights reserved.
Schweitzer Classification
Persons
Daniel C. M. de Oliveira obtained his Ph.D. in Systems and Computation Engineering at COPPE/Federal University of Rio de Janeiro, Brazil, in 2012. His current research interests include scientific workflows, provenance, cloud computing, data scalable and intensive computing, high performance computing, and distributed and parallel databases. He serves or served on the program committee of major international and national conferences (VLDB17, IPAW16 and 18, SBBD 15-18, etc.) and is a member of IEEE, ACM, and the Brazilian Computer Society. In 2016, he received the Young Scientist scholarship from the State Agency for Research Financing of Rio de Janeiro, FAPERJ, and the Level 2 research productivity grant from CNPq. He supervised 5 doctoral theses and 13 master's dissertations and coordinated projects funded by CNPq and FAPERJ. He has 2,077 citations in Google Scholar, an h-index of 22 and 96 articles listed in DBLP.
Ji Liu is at Microsoft Research Inria Joint Centre and Zenith team. The latter is part of INRIA Sophia-Antipolis Mediterranee and LIRMM at Montpellier. His research interests include scientific workflow, big data, Cloud computing, and multisite management. He graduated from the Xidian University in 2011. Then, he obtained his master's degree (diplôme d'ingénieur) from Télécom SudParis in 2013 and Ph.D. in 2016 from University of Montpellier.
Esther Pacitti is a professor of computer science at University of Montpellier. She is a senior researcher and co-head of the Zenith team at LIRMM, pursuing research in distributed data management. Previously, she was an assistant professor at University of Nantes (2002-2009) and a member of Atlas INRIA team. She obtained her "Habilitation à Diriger les Recherches" (HDR) degree in 2008 on the topic of data replication on different contexts (data warehouses, clusters and peer-to-peer systems). Since 2004 she has served or is serving as program committee member of major international conferences (VLDB, SIGMOD, CIKM, etc.) and has edited and co-authored several books. She has also published a significant amount of technical papers and journal papers in well-known international conferences and journals.
H. V. Jagadish is Bernard A Galler Collegiate Professor of Electrical Engineering and Computer Science, and Distinguished Scientist at the Institute for Data Science, at the University of Michigan in Ann Arbor. Prior to 1999, he was Head of the Database Research Department at AT&T Labs, Florham Park, NJ. Professor Jagadish is well known for his broad-ranging research on information management, and has approximately 200 major papers and 37 patents. He is a fellow of the ACM, "The First Society in Computing," (since 2003) and serves on the board of the Computing Research Association (since 2009). He has been an Associate Editor for the ACM Transactions on Database Systems (1992-1995), Program Chair of the ACM SIGMOD annual conference (1996), Program Chair of the ISMB conference (2005), a trustee of the VLDB (Very Large DataBase) foundation (2004-2009), Founding Editor-in-Chief of the Proceedings of the VLDB Endowment (2008-2014), and Program Chair of the VLDB Conference (2014). Among his many awards, he won the ACM SIGMOD Contributions Award in 2013 and the David E Liddle Research Excellence Award (at the University of Michigan) in 2008.
Ji Liu is at Microsoft Research Inria Joint Centre and Zenith team. The latter is part of INRIA Sophia-Antipolis Mediterranee and LIRMM at Montpellier. His research interests include scientific workflow, big data, Cloud computing, and multisite management. He graduated from the Xidian University in 2011. Then, he obtained his master's degree (diplôme d'ingénieur) from Télécom SudParis in 2013 and Ph.D. in 2016 from University of Montpellier.
Esther Pacitti is a professor of computer science at University of Montpellier. She is a senior researcher and co-head of the Zenith team at LIRMM, pursuing research in distributed data management. Previously, she was an assistant professor at University of Nantes (2002-2009) and a member of Atlas INRIA team. She obtained her "Habilitation à Diriger les Recherches" (HDR) degree in 2008 on the topic of data replication on different contexts (data warehouses, clusters and peer-to-peer systems). Since 2004 she has served or is serving as program committee member of major international conferences (VLDB, SIGMOD, CIKM, etc.) and has edited and co-authored several books. She has also published a significant amount of technical papers and journal papers in well-known international conferences and journals.
H. V. Jagadish is Bernard A Galler Collegiate Professor of Electrical Engineering and Computer Science, and Distinguished Scientist at the Institute for Data Science, at the University of Michigan in Ann Arbor. Prior to 1999, he was Head of the Database Research Department at AT&T Labs, Florham Park, NJ. Professor Jagadish is well known for his broad-ranging research on information management, and has approximately 200 major papers and 37 patents. He is a fellow of the ACM, "The First Society in Computing," (since 2003) and serves on the board of the Computing Research Association (since 2009). He has been an Associate Editor for the ACM Transactions on Database Systems (1992-1995), Program Chair of the ACM SIGMOD annual conference (1996), Program Chair of the ISMB conference (2005), a trustee of the VLDB (Very Large DataBase) foundation (2004-2009), Founding Editor-in-Chief of the Proceedings of the VLDB Endowment (2008-2014), and Program Chair of the VLDB Conference (2014). Among his many awards, he won the ACM SIGMOD Contributions Award in 2013 and the David E Liddle Research Excellence Award (at the University of Michigan) in 2008.
Content
- Preface
- Acknowledgments
- Overview
- Background Knowledge
- Workflow Execution in a Single-Site Cloud
- Workflow Execution in a Multi-Site Cloud
- Workflow Execution in DISC Environments
- Conclusion
- Bibliography
- Authors' Biographies
- Acknowledgments
- Overview
- Background Knowledge
- Workflow Execution in a Single-Site Cloud
- Workflow Execution in a Multi-Site Cloud
- Workflow Execution in DISC Environments
- Conclusion
- Bibliography
- Authors' Biographies