Data Engineering on Azure

Name: Data Engineering on Azure
Brand: Manning
Price: 37.67 EUR
Availability: OnlineOnly

Vlad Riscutia(Autor*in)

Manning (Verlag)

1. Auflage

Erschienen am 21. September 2021

336 Seiten

E-Book

ePUB mit Adobe-DRM

Systemvoraussetzungen

978-1-63835-691-2 (ISBN)

37,67 €inkl. 7% MwSt.

Systemvoraussetzungen

für ePUB mit Adobe-DRM

E-Book Einzellizenz

Als Download verfügbar

Beschreibung

Build a data platform to the industry-leading standards set by Microsoft's own infrastructure.

Summary
In Data Engineering on Azure you will learn how to:

Pick the right Azure services for different data scenarios
Manage data inventory
Implement production quality data modeling, analytics, and machine learning workloads
Handle data governance
Using DevOps to increase reliability
Ingesting, storing, and distributing data
Apply best practices for compliance and access control

Data Engineering on Azure reveals the data management patterns and techniques that support Microsoft's own massive data infrastructure. Author Vlad Riscutia, a data engineer at Microsoft, teaches you to bring an engineering rigor to your data platform and ensure that your data prototypes function just as well under the pressures of production. You'll implement common data modeling patterns, stand up cloud-native data platforms on Azure, and get to grips with DevOps for both analytics and machine learning.

Purchase of the print book includes a free eBook in PDF, Kindle, and ePub formats from Manning Publications.

About the technology
Build secure, stable data platforms that can scale to loads of any size. When a project moves from the lab into production, you need confidence that it can stand up to real-world challenges. This book teaches you to design and implement cloud-based data infrastructure that you can easily monitor, scale, and modify.

About the book
In Data Engineering on Azure you'll learn the skills you need to build and maintain big data platforms in massive enterprises. This invaluable guide includes clear, practical guidance for setting up infrastructure, orchestration, workloads, and governance. As you go, you'll set up efficient machine learning pipelines, and then master time-saving automation and DevOps solutions. The Azure-based examples are easy to reproduce on other cloud platforms.

What's inside

Data inventory and data governance
Assure data quality, compliance, and distribution
Build automated pipelines to increase reliability
Ingest, store, and distribute data
Production-quality data modeling, analytics, and machine learning

About the reader
For data engineers familiar with cloud computing and DevOps.

About the author
Vlad Riscutia is a software architect at Microsoft.

Table of Contents

1 Introduction
PART 1 INFRASTRUCTURE
2 Storage
3 DevOps
4 Orchestration
PART 2 WORKLOADS
5 Processing
6 Analytics
7 Machine learning
PART 3 GOVERNANCE
8 Metadata
9 Data quality
10 Compliance
11 Distributing data

Weitere Details

Weitere Ausgaben

Person

Inhalt

Intro
inside front cover
Data Platform Architecture
Data Engineering on Azure
Copyright
dedication
brief contents
contents
front matter
preface
acknowledgments
about this book
about the author
about the cover illustration
1 Introduction
1.1 What is data engineering?
1.2 Who this book is for
1.3 What is a data platform?
1.3.1 Anatomy of a data platform
1.3.2 Infrastructure as code, codeless infrastructure
1.4 Building in the cloud
1.4.1 IaaS, PaaS, SaaS
1.4.2 Network, storage, compute
1.4.3 Getting started with Azure
1.4.4 Interacting with Azure
1.5 Implementing an Azure data platform
Summary
Part 1 Infrastructure
2 Storage
2.1 Storing data in a data platform
2.1.1 Storing data across multiple data fabrics
2.1.2 Having a single source of truth
2.2 Introducing Azure Data Explorer
2.2.1 Deploying an Azure Data Explorer cluster
2.2.2 Using Azure Data Explorer
2.2.3 Working around query limits
2.3 Introducing Azure Data Lake Storage
2.3.1 Creating an Azure Data Lake Storage account
2.3.2 Using Azure Data Lake Storage
2.3.3 Integrating with Azure Data Explorer
2.4 Ingesting data
2.4.1 Ingestion frequency
2.4.2 Load type
2.4.3 Restatements and reloads
Summary
3 DevOps
3.1 What is DevOps?
3.1.1 DevOps in data engineering
3.2 Introducing Azure DevOps
3.2.1 Using the az azure-devops extension
3.3 Deploying infrastructure
3.3.1 Exporting an Azure Resource Manager template
3.3.2 Creating Azure DevOps service connections
3.3.3 Deploying Azure Resource Manager templates
3.3.4 Understanding Azure Pipelines
3.4 Deploying analytics
3.4.1 Using Azure DevOps marketplace extensions
3.4.2 Storing everything in Git
deploying everything automatically
Summary
4 Orchestration
4.1 Ingesting the Bing COVID-19 open dataset
4.2 Introducing Azure Data Factory
4.2.1 Setting up the data source
4.2.2 Setting up the data sink
4.2.3 Setting up the pipeline
4.2.4 Setting up a trigger
4.2.5 Orchestrating with Azure Data Factory
4.3 DevOps for Azure Data Factory
4.3.1 Deploying Azure Data Factory from Git
4.3.2 Setting up access control
4.3.3 Deploying the production data factory
4.3.4 DevOps for the Azure Data Factory recap
4.4 Monitoring with Azure Monitor
Summary
Part 2 Workloads
5 Processing
5.1 Data modeling techniques
5.1.1 Normalization and denormalization
5.1.2 Data warehousing
5.1.3 Semistructured data
5.1.4 Data modeling recap
5.2 Identity keyrings
5.2.1 Building an identity keyring
5.2.2 Understanding keyrings
5.3 Timelines
5.3.1 Building a timeline view
5.3.2 Using timelines
5.4 Continuous data processing
5.4.1 Tracking processing functions in Git
5.4.2 Keyring building in Azure Data Factory
5.4.3 Scaling out
Summary
6 Analytics
6.1 Structuring storage
6.1.1 Providing development data
6.1.2 Replicating production data
6.1.3 Providing read-only access to the production data
6.1.4 Storage structure recap
6.2 Analytics workflow
6.2.1 Prototyping
6.2.2 Development and user acceptance testing
6.2.3 Production
6.2.4 Analytics workflow recap
6.3 Self-serve data movement
6.3.1 Support model
6.3.2 Data contracts
6.3.3 Pipeline validation
6.3.4 Postmortems
6.3.5 Self-serve data movement recap
Summary
7 Machine learning
7.1 Training a machine learning model
7.1.1 Training a model using scikit-learn
7.1.2 High spender model implementation
7.2 Introducing Azure Machine Learning
7.2.1 Creating a workspace
7.2.2 Creating an Azure Machine Learning compute target
7.2.3 Setting up Azure Machine Learning storage
7.2.4 Running ML in the cloud
7.2.5 Azure Machine Learning recap
7.3 MLOps
7.3.1 Deploying from Git
7.3.2 Storing pipeline IDs
7.3.3 DevOps for Azure Machine Learning recap
7.4 Orchestrating machine learning
7.4.1 Connecting Azure Data Factory with Azure Machine Learning
7.4.2 Machine learning orchestration
7.4.3 Orchestrating recap
Summary
Part 3 Governance
8 Metadata
8.1 Making sense of the data
8.2 Introducing Azure Purview
8.3 Maintaining a data inventory
8.3.1 Setting up a scan
8.3.2 Browsing the data dictionary
8.3.3 Data dictionary recap
8.4 Managing a data glossary
8.4.1 Adding a new glossary term
8.4.2 Curating terms
8.4.3 Custom templates and bulk import
8.4.4 Data glossary recap
8.5 Understanding Azure Purview's advanced features
8.5.1 Tracking lineage
8.5.2 Classification rules
8.5.3 REST API
8.5.4 Advanced features recap
Summary
9 Data quality
9.1 Testing data
9.1.1 Availability tests
9.1.2 Correctness tests
9.1.3 Completeness tests
9.1.4 Detecting anomalies
9.1.5 Testing data recap
9.2 Running data quality checks
9.2.1 Testing using Azure Data Factory
9.2.2 Executing tests
9.2.3 Creating and using a template
9.2.4 Running data quality checks recap
9.3 Scaling out data testing
9.3.1 Supporting multiple data fabrics
9.3.2 Testing at rest and during movement
9.3.3 Authoring tests
9.3.4 Storing tests and results
Summary
10 Compliance
10.1 Data classification
10.1.1 Feature data
10.1.2 Telemetry
10.1.3 User data
10.1.4 User-owned data
10.1.5 Business data
10.1.6 Data classification recap
10.2 Changing classification through processing
10.2.1 Aggregation
10.2.2 Anonymization
10.2.3 Pseudonymization
10.2.4 Masking
10.2.5 Processing classification changes recap
10.3 Implementing an access model
10.3.1 Security groups
10.3.2 Securing Azure Data Explorer
10.3.3 Access model recap
10.4 Complying with GDPR and other considerations
10.4.1 Data handling
10.4.2 Data subject requests
10.4.3 Other considerations
Summary
11 Distributing data
11.1 Data distribution overview
11.2 Building a data API
11.2.1 Introducing Azure Cosmos DB
11.2.2 Populating the Cosmos DB collection
11.2.3 Retrieving data
11.2.4 Data API recap
11.3 Serving machine learning
11.4 Sharing data for bulk copy
11.4.1 Separating compute resources
11.4.2 Introducing Azure Data Share
11.4.3 Sharing data for bulk copy recap
11.5 Data sharing best practices
Summary
Appendix A. Azure services
Azure Storage
Azure SQL
Azure Synapse Analytics
Azure Data Explorer
Azure Databricks
Azure Cosmos DB
Appendix B. KQL quick reference
Common query reference
SQL to KQL
Appendix C. Running code samples
index
inside back cover
MLOps

Systemvoraussetzungen

Als PDF speichern Als Link merken