Learning Cascading
Build Reliable, Robust, and High-Performance Big Data Applications Using the Cascading Application Development Efficiently
1. Auflage
Erschienen am 29. Mai 2015
276 Seiten
978-1-78528-523-3 (ISBN)
Beschreibung
Cascading is open source software that is used to create and execute complex data processing workflows on big data clusters. The book starts by explaining how Cascading relates to core big data technologies such as Hadoop MapReduce. Having instilled an understanding of the technology, the book provides a comprehensive introduction to the Cascading paradigm and its components using code examples. You will not only learn more advanced Cascading features, you will also write code to utilize them. Furthermore, you will gain in-depth knowledge of how to efficiently optimize a Cascading application. To deepen your knowledge and experience with Cascading, you will work through a real-life case study using Natural Language Processing to perform text analysis and search on large volumes of unstructured text. Throughout the book, you will receive expert advice on how to use the portions of the product that are undocumented or have limited documentation. By the end of the book, you will be able to build practical Cascading applications.
Alle Preise
Weitere Details
Weitere Ausgaben
Inhalt
- Cover
- Copyright
- Credits
- Foreword
- About the Authors
- About the Reviewers
- www.PacktPub.com
- Table of Contents
- Preface
- Chapter 1: The Big Data Core Technology Stack
- Reviewing Hadoop
- Hadoop architecture
- HDFS - the Hadoop Distributed File System
- The NameNode
- The secondary NameNode
- DataNodes
- MapReduce execution framework
- The JobTracker
- The TaskTracker
- Hadoop jobs
- Distributed cache
- Counters
- YARN - MapReduce version 2
- A simple MapReduce job
- Beyond MapReduce
- The Cascading framework
- The execution graph and flow planner
- How Cascading produces MapReduce jobs
- Summary
- Chapter 2: Cascading Basics in Detail
- Understanding common Cascading themes
- Data flows as processes
- Understanding how Cascading represents records
- Using tuples and defining fields
- Using a Fields object, named field groups and selectors
- Data typing and coercion
- Defining schemes
- Schemes in detail
- TupleEntry
- Understanding how Cascading controls data flow
- Using pipes
- Creating and chaining
- Pipe operations
- Each
- Splitting
- GroupBy and sorting
- Every
- Merging and joining
- The Merge pipe
- The join pipes - CoGroup and HashJoin
- CoGroup
- HashJoin
- Default output selectors
- Using taps
- Flow
- FlowConnector
- Cascades
- Local and Hadoop modes
- Common errors
- Putting it all together
- Summary
- Chapter 3: Understanding Custom Operations
- Understanding operations
- Operations and fields
- The Operation class and interface hierarchy
- The basic operation lifecycle
- Contexts
- FlowProcess
- OperationCall&Context&
- An operation processing sequence and its methods
- Operation types
- Each operations
- Every operations
- Buffers
- Assertions
- Summary
- Chapter 4: Creating Custom Operations
- Writing custom operations
- Writing a filter
- Writing a function
- Writing an aggregator
- Writing a custom assertion
- Writing a buffer
- Identifying common use cases for custom operations
- Putting it all together
- Summary
- Chapter 5: Code Reuse and Integration
- Creating and using subassemblies
- Built-in subassemblies
- Creating a new custom subassembly
- Using custom subassemblies
- Using cascades
- Building a complex workflow using cascades
- Skipping a flow in a cascade
- Intermediate file management
- Dynamically controlling flows
- Instrumentation and counters
- Using counters to control flow
- Using existing MapReduce jobs
- The FlowDef fluent interface
- Integrating external components
- Flow and cascade events
- Using external JAR files
- Using Cascading as insulation from big data migrations and upgrades
- Summary
- Chapter 6: Testing a Cascading Application
- Debugging a Cascading application
- Getting your environment ready for debugging
- Using Cascading local mode debugging
- Setting up Eclipse
- Remote debugging
- Using assertions
- The Debug() filter
- Managing exceptions with traps
- Checkpoints
- Managing bad data
- Viewing flow sequencing using DOT files
- Testing strategies
- Unit testing and JUnit
- Mocking
- Integration testing
- Load and performance testing
- Summary
- Chapter 7: Optimizing the Performance of a Cascading Application
- Optimizing performance
- Optimizing Cascading
- Optimizing Hadoop
- A note about the effective use of checkpoints
- Summary
- Chapter 8: Creating a Real-world Application in Cascading
- Project description - Business Intelligence case study on monitoring the competition
- Project scope - understanding requirements
- Understanding the project domain - text analytics and natural language processing (NLP)
- Conducting a simple named entity extraction
- Defining the project - the Cascading development methodology
- Project roles and responsibilities
- Conducting data analysis
- Performing functional decomposition
- Designing the process and components
- Creating and integrating the operations
- Creating and using subassemblies
- Building the workflow
- Building flows
- Managing the context
- Building the cascade
- Designing the test plan
- Performing a unit test
- Performing an integration test
- Performing a cluster test
- Performing a full load test
- Refining and adjusting
- Software packaging and delivery to the cluster
- Next steps
- Summary
- Chapter 9: Planning for Future Growth
- Finding online resources
- Using other Cascading tools
- Lingual
- Pattern
- Driven
- Fluid
- Load
- Multitool
- Support for other languages
- Hortonworks
- Custom Taps
- Cascading serializers
- Java open source mock frameworks
- Summary
- Appendix: Downloadable Software
- Contents
- Installing and using
- Index
