Apache Flume: Distributed Log Collection for Hadoop
Design and implement a series of Flume agents to send streamed data into Hadoop
2nd Edition
Published on 7. October 2025
178 pages
978-1-78439-914-6 (ISBN)
Description
Key Features
Book DescriptionIf you are a Hadoop programmer who wants to learn about Flume to be able to move datasets into Hadoop in a timely and replicable manner, then this book is ideal for you. No prior knowledge about Apache Flume is necessary, but a basic knowledge of Hadoop and the Hadoop File System (HDFS) is assumed.What you will learn
Understand the Flume architecture, and also how to download and install open source Flume from Apache
Follow along a detailed example of transporting weblogs in Near Real Time (NRT) to Kibana/Elasticsearch and archival in HDFS
Learn tips and tricks for transporting logs and data in your production environment
Understand and configure the Hadoop File System (HDFS) Sink
Use a morphlinebacked Sink to feed data into Solr
Create redundant data flows using sink groups
Configure and use various sources to ingest data
Inspect data records and move them between multiple destinations based on payload content
Transform data enroute to Hadoop and monitor your data flows
Who this book is for
Book DescriptionIf you are a Hadoop programmer who wants to learn about Flume to be able to move datasets into Hadoop in a timely and replicable manner, then this book is ideal for you. No prior knowledge about Apache Flume is necessary, but a basic knowledge of Hadoop and the Hadoop File System (HDFS) is assumed.What you will learn
Understand the Flume architecture, and also how to download and install open source Flume from Apache
Follow along a detailed example of transporting weblogs in Near Real Time (NRT) to Kibana/Elasticsearch and archival in HDFS
Learn tips and tricks for transporting logs and data in your production environment
Understand and configure the Hadoop File System (HDFS) Sink
Use a morphlinebacked Sink to feed data into Solr
Create redundant data flows using sink groups
Configure and use various sources to ingest data
Inspect data records and move them between multiple destinations based on payload content
Transform data enroute to Hadoop and monitor your data flows
Who this book is for
All prices
More details
Edition
2nd Revised edition
Language
English
Place of publication
Birmingham
United Kingdom
Target group
Professional and scholarly
US School Grade: College Graduate Student
Edition type
Revised edition
File size
2,57 MB
ISBN-13
978-1-78439-914-6 (9781784399146)
Copyright in bibliographic data and cover images is held by Nielsen Book Services Limited or by the publishers or by their respective licensors: all rights reserved.
Schweitzer Classification
Other editions
Additional editions
Book
02/2015
Packt Publishing
€47.10
Shipment within 3-4 weeks
Person
Hoffman Steven :
Steve Hoffman has 32 years of experience in software development, ranging from embedded software development to the design and implementation of large-scale, service-oriented, object-oriented systems. For the last 5 years, he has focused on infrastructure as code, including automated Hadoop and HBase implementations and data ingestion using Apache Flume. Steve holds a BS in computer engineering from the University of Illinois at Urbana-Champaign and an MS in computer science from DePaul University. He is currently a senior principal engineer at Orbitz Worldwide (http://orbitz.com/). More information on Steve can be found at http://bit.ly/bacoboy and on Twitter at @bacoboy. This is the first update to Steve's first book, Apache Flume: Distributed Log Collection for Hadoop, Packt Publishing.
Steve Hoffman has 32 years of experience in software development, ranging from embedded software development to the design and implementation of large-scale, service-oriented, object-oriented systems. For the last 5 years, he has focused on infrastructure as code, including automated Hadoop and HBase implementations and data ingestion using Apache Flume. Steve holds a BS in computer engineering from the University of Illinois at Urbana-Champaign and an MS in computer science from DePaul University. He is currently a senior principal engineer at Orbitz Worldwide (http://orbitz.com/). More information on Steve can be found at http://bit.ly/bacoboy and on Twitter at @bacoboy. This is the first update to Steve's first book, Apache Flume: Distributed Log Collection for Hadoop, Packt Publishing.
Content
Table of Contents
Overview and Architecture
Flume Quick Start
Channels
Sinks and Sink Processors
Sources and Channel Selectors
Interceptors ETL and Routing
Examples
Monitoring Flume
3 D Registration and Visualization
Overview and Architecture
Flume Quick Start
Channels
Sinks and Sink Processors
Sources and Channel Selectors
Interceptors ETL and Routing
Examples
Monitoring Flume
3 D Registration and Visualization
