Apache Flume: Distributed Log Collection for Hadoop - Second Edition
Design and Implement a Series of Flume Agents to Send Streamed Data into Hadoop
1. Auflage
Erschienen am 25. Februar 2015
178 Seiten
978-1-78439-914-6 (ISBN)
Beschreibung
Apache Flume is a distributed, reliable, and available service used to efficiently collect, aggregate, and move large amounts of log data. It is used to stream logs from application servers to HDFS for ad hoc analysis.This book starts with an architectural overview of Flume and its logical components. It explores channels, sinks, and sink processors, followed by sources and channels. By the end of this book, you will be fully equipped to construct a series of Flume agents to dynamically transport your stream data and logs from your systems into Hadoop.A step-by-step book that guides you through the architecture and components of Flume covering different approaches, which are then pulled together as a real-world, end-to-end use case, gradually going from the simplest to the most advanced features.
Alle Preise
Weitere Details
Weitere Ausgaben
Inhalt
- Intro
- Apache Flume: Distributed Log Collection for Hadoop Second Edition
- Table of Contents
- Apache Flume: Distributed Log Collection for Hadoop Second Edition
- Credits
- About the Author
- About the Reviewers
- www.PacktPub.com
- Support files, eBooks, discount offers, and more
- Why subscribe?
- Free access for Packt account holders
- Preface
- What this book covers
- What you need for this book
- Who this book is for
- Conventions
- Reader feedback
- Customer support
- Downloading the example code
- Errata
- Piracy
- Questions
- 1. Overview and Architecture
- Flume 0.9
- Flume 1.X (Flume-NG)
- The problem with HDFS and streaming data/logs
- Sources, channels, and sinks
- Flume events
- Interceptors, channel selectors, and sink processors
- Tiered data collection (multiple flows and/or agents)
- The Kite SDK
- Summary
- 2. A Quick Start Guide to Flume
- Downloading Flume
- Flume in Hadoop distributions
- An overview of the Flume configuration file
- Starting up with "Hello, World!
- Summary
- 3. Channels
- The memory channel
- The file channel
- Spillable Memory Channel
- Summary
- 4. Sinks and Sink Processors
- HDFS sink
- Path and filename
- File rotation
- Compression codecs
- Event Serializers
- Text output
- Text with headers
- Apache Avro
- User-provided Avro schema
- File type
- SequenceFile
- DataStream
- CompressedStream
- Timeouts and workers
- Sink groups
- Load balancing
- Failover
- MorphlineSolrSink
- Morphline configuration files
- Typical SolrSink configuration
- Sink configuration
- ElasticSearchSink
- LogStash Serializer
- Dynamic Serializer
- Summary
- 5. Sources and Channel Selectors
- The problem with using tail
- The Exec source
- Spooling Directory Source
- Syslog sources
- The syslog UDP source
- The syslog TCP source
- The multiport syslog TCP source
- JMS source
- Channel selectors
- Replicating
- Multiplexing
- Summary
- 6. Interceptors, ETL, and Routing
- Interceptors
- Timestamp
- Host
- Static
- Regular expression filtering
- Regular expression extractor
- Morphline interceptor
- Custom interceptors
- The plugins directory
- Tiering flows
- The Avro source/sink
- Compressing Avro
- SSL Avro flows
- The Thrift source/sink
- Using command-line Avro
- The Log4J appender
- The Log4J load-balancing appender
- The embedded agent
- Configuration and startup
- Sending data
- Shutdown
- Routing
- Summary
- 7. Putting It All Together
- Web logs to searchable UI
- Setting up the web server
- Configuring log rotation to the spool directory
- Setting up the target - Elasticsearch
- Setting up Flume on collector/relay
- Setting up Flume on the client
- Creating more search fields with an interceptor
- Setting up a better user interface - Kibana
- Archiving to HDFS
- Summary
- 8. Monitoring Flume
- Monitoring the agent process
- Monit
- Nagios
- Monitoring performance metrics
- Ganglia
- Internal HTTP server
- Custom monitoring hooks
- Summary
- 9. There Is No Spoon - the Realities of Real-time Distributed Data Collection
- Transport time versus log time
- Time zones are evil
- Capacity planning
- Considerations for multiple data centers
- Compliance and data expiry
- Summary
- Index
