Designing Cloud Data Platforms

Name: Designing Cloud Data Platforms
Brand: Manning
Price: 44.8 EUR
Availability: OnlineOnly

Danil Zburivsky Lynda Partner(Autor*in)

Manning (Verlag)

1. Auflage

Erschienen am 17. März 2021

336 Seiten

E-Book

ePUB mit Adobe-DRM

Systemvoraussetzungen

978-1-63835-096-5 (ISBN)

44,80 €inkl. 7% MwSt.

Systemvoraussetzungen

für ePUB mit Adobe-DRM

E-Book Einzellizenz

Als Download verfügbar

Beschreibung

In Designing Cloud Data Platforms, Danil Zburivsky and Lynda Partner reveal a six-layer approach that increases flexibility and reduces costs. Discover patterns for ingesting data from a variety of sources, then learn to harness pre-built services provided by cloud vendors.

Summary
Centralized data warehouses, the long-time defacto standard for housing data for analytics, are rapidly giving way to multi-faceted cloud data platforms. Companies that embrace modern cloud data platforms benefit from an integrated view of their business using all of their data and can take advantage of advanced analytic practices to drive predictions and as yet unimagined data services. Designing Cloud Data Platforms is a hands-on guide to envisioning and designing a modern scalable data platform that takes full advantage of the flexibility of the cloud. As you read, you'll learn the core components of a cloud data platform design, along with the role of key technologies like Spark and Kafka Streams. You'll also explore setting up processes to manage cloud-based data, keep it secure, and using advanced analytic and BI tools to analyze it.

Purchase of the print book includes a free eBook in PDF, Kindle, and ePub formats from Manning Publications.

About the technology
Well-designed pipelines, storage systems, and APIs eliminate the complicated scaling and maintenance required with on-prem data centers. Once you learn the patterns for designing cloud data platforms, you'll maximize performance no matter which cloud vendor you use.

About the book
In Designing Cloud Data Platforms, Danil Zburivsky and Lynda Partner reveal a six-layer approach that increases flexibility and reduces costs. Discover patterns for ingesting data from a variety of sources, then learn to harness pre-built services provided by cloud vendors.

What's inside
Best practices for structured and unstructured data sets
Cloud-ready machine learning tools
Metadata and real-time analytics
Defensive architecture, access, and security

About the reader
For data professionals familiar with the basics of cloud computing, and Hadoop or Spark.

About the author
Danil Zburivsky has over 10 years of experience designing and supporting large-scale data infrastructure for enterprises across the globe. Lynda Partner is the VP of Analytics-as-a-Service at Pythian, and has been on the business side of data for over 20 years.

Table of Contents
1 Introducing the data platform
2 Why a data platform and not just a data warehouse
3 Getting bigger and leveraging the Big 3: Amazon, Microsoft Azure, and Google
4 Getting data into the platform
5 Organizing and processing data
6 Real-time data processing and analytics
7 Metadata layer architecture
8 Schema management
9 Data access and security
10 Fueling business value with data platforms

Weitere Details

Weitere Ausgaben

Personen

Inhalt

Intro
inside front cover
Designing Cloud Data Platforms
Copyright
brief contents
contents
front matter
preface
acknowledgments
about this book
Who should read this book
How this book is organized: A roadmap
About the code
liveBook discussion forum
about the authors
about the cover illustration
1 Introducing the data platform
1.1 The trends behind the change from data warehouses to data platforms
1.2 Data warehouses struggle with data variety, volume, and velocity
1.2.1 Variety
1.2.2 Volume
1.2.3 Velocity
1.2.4 All the V's at once
1.3 Data lakes to the rescue?
1.4 Along came the cloud
1.5 Cloud, data lakes, and data warehouses: The emergence of cloud data platforms
1.6 Building blocks of a cloud data platform
1.6.1 Ingestion layer
1.6.2 Storage layer
1.6.3 Processing layer
1.6.4 Serving layer
1.7 How the cloud data platform deals with the three V's
1.7.1 Variety
1.7.2 Volume
1.7.3 Velocity
1.7.4 Two more V's
1.8 Common use cases
Summary
2 Why a data platform and not just a data warehouse
2.1 Cloud data platforms and cloud data warehouses: The practical aspects
2.1.1 A closer look at the data sources
2.1.2 An example cloud data warehouse-only architecture
2.1.3 An example cloud data platform architecture
2.2 Ingesting data
2.2.1 Ingesting data directly into Azure Synapse
2.2.2 Ingesting data into an Azure data platform
2.2.3 Managing changes in upstream data sources
2.3 Processing data
2.3.1 Processing data in the warehouse
2.3.2 Processing data in the data platform
2.4 Accessing data
2.5 Cloud cost considerations
Summary
2.6 Exercise answers
3 Getting bigger and leveraging the Big 3: Amazon, Microsoft Azure, and Google
3.1 Cloud data platform layered architecture
3.1.1 Data ingestion layer
3.1.2 Fast and slow storage
3.1.3 Processing layer
3.1.4 Technical metadata layer
3.1.5 The serving layer and data consumers
3.1.6 Orchestration and ETL overlay layers
3.2 The importance of layers in a data platform architecture
3.3 Mapping cloud data platform layers to specific tools
3.3.1 AWS
3.3.2 Google Cloud
3.3.3 Azure
3.4 Open source and commercial alternatives
3.4.1 Batch data ingestion
3.4.2 Streaming data ingestion and real-time analytics
3.4.3 Orchestration layer
Summary
3.5 Exercise answers
4 Getting data into the platform
4.1 Databases, files, APIs, and streams
4.1.1 Relational databases
4.1.2 Files
4.1.3 SaaS data via API
4.1.4 Streams
4.2 Ingesting data from relational databases
4.2.1 Ingesting data from RDBMSs using a SQL interface
4.2.2 Full-table ingestion
4.2.3 Incremental table ingestion
4.2.4 Change data capture (CDC)
4.2.5 CDC vendors overview
4.2.6 Data type conversion
4.2.7 Ingesting data from NoSQL databases
4.2.8 Capturing important metadata for RDBMS or NoSQL ingestion pipelines
4.3 Ingesting data from files
4.3.1 Tracking ingested files
4.3.2 Capturing file ingestion metadata
4.4 Ingesting data from streams
4.4.1 Differences between batch and streaming ingestion
4.4.2 Capturing streaming pipeline metadata
4.5 Ingesting data from SaaS applications
4.5.1 No standard approach to API design
4.5.2 No standard way to deal with full vs. incremental data exports
4.5.3 Resulting data is typically highly nested JSON
4.6 Network and security considerations for data ingestion into the cloud
4.6.1 Connecting other networks to your cloud data platform
Summary
4.7 Exercise answers
5 Organizing and processing data
5.1 Processing as a separate layer in the data platform
5.2 Data processing stages
5.3 Organizing your cloud storage
5.3.1 Cloud storage containers and folders
5.4 Common data processing steps
5.4.1 File format conversion
5.4.2 Data deduplication
5.4.3 Data quality checks
5.5 Configurable pipelines
Summary
5.6 Exercise answers
6 Real-time data processing and analytics
6.1 Real-time ingestion vs. real-time processing
6.2 Use cases for real-time data processing
6.2.1 Retail use case: Real-time ingestion
6.2.2 Online gaming use case: Real-time ingestion and real-time processing
6.2.3 Summary of real-time ingestion vs. real-time processing
6.3 When should you use real-time ingestion and/or real-time processing?
6.4 Organizing data for real-time use
6.4.1 The anatomy of fast storage
6.4.2 How does fast storage scale?
6.4.3 Organizing data in the real-time storage
6.5 Common data transformations in real time
6.5.1 Causes of duplicates in real-time systems
6.5.2 Deduplicating data in real-time systems
6.5.3 Converting message formats in real-time pipelines
6.5.4 Real-time data quality checks
6.5.5 Combining batch and real-time data
6.6 Cloud services for real-time data processing
6.6.1 AWS real-time processing services
6.6.2 Google Cloud real-time processing services
6.6.3 Azure real-time processing services
Summary
6.7 Exercise answers
7 Metadata layer architecture
7.1 What we mean by metadata
7.1.1 Business metadata
7.1.2 Data platform internal metadata or "pipeline metadata"
7.2 Taking advantage of pipeline metadata
7.3 Metadata model
7.3.1 Metadata domains
7.4 Metadata layer implementation options
7.4.1 Metadata layer as a collection of configuration files
7.4.2 Metadata database
7.4.3 Metadata API
7.5 Overview of existing solutions
7.5.1 Cloud metadata services
7.5.2 Open source metadata layer implementations
Summary
7.6 Exercise answers
8 Schema management
8.1 Why schema management
8.1.1 Schema changes in a traditional data warehouse architecture
8.1.2 Schema-on-read approach
8.2 Schema-management approaches
8.2.1 Schema as a contract
8.2.2 Schema management in the data platform
8.2.3 Monitoring schema changes
8.3 Schema Registry Implementation
8.3.1 Apache Avro schemas
8.3.2 Existing Schema Registry implementations
8.3.3 Schema Registry as part of a Metadata layer
8.4 Schema evolution scenarios
8.4.1 Schema compatibility rules
8.4.2 Schema evolution and data transformation pipelines
8.5 Schema evolution and data warehouses
8.5.1 Schema-management features of cloud data warehouses
Summary
8.6 Exercise answers
9 Data access and security
9.1 Different types of data consumers
9.2 Cloud data warehouses
9.2.1 AWS Redshift
9.2.2 Azure Synapse
9.2.3 Google BigQuery
9.2.4 Choosing the right data warehouse
9.3 Application data access
9.3.1 Cloud relational databases
9.3.2 Cloud key/value data stores
9.3.3 Full-text search services
9.3.4 In-memory cache
9.4 Machine learning on the data platform
9.4.1 Machine learning model lifecycle on a cloud data platform
9.4.2 ML cloud collaboration tools
9.5 Business intelligence and reporting tools
9.5.1 Traditional BI tools and cloud data platform integration
9.5.2 Using Excel as a BI tool
9.5.3 BI tools that are external to the cloud provider
9.6 Data security
9.6.1 Users, groups, and roles
9.6.2 Credentials and configuration management
9.6.3 Data encryption
9.6.4 Network boundaries
Summary
9.7 Exercise Answers
10 Fueling business value with data platforms
10.1 Why you need a data strategy
10.2 The analytics maturity journey
10.2.1 SEE: Getting insights from data
10.2.2 PREDICT: Using data to predict what to do
10.2.3 DO: Making your analytics actionable
10.2.4 CREATE: Going beyond analytics into products
10.3 The data platform: The engine that powers analytics maturity
10.4 Platform project stoppers
10.4.1 Time does indeed kill
10.4.2 User adoption
10.4.3 User trust and the need for data governance
10.4.4 Operating in a platform silo
10.4.5 The dollar dance
Summary
index
inside back cover

Systemvoraussetzungen

Als PDF speichern Als Link merken