Introduction

A long time ago, I was sitting at my desk happily coding my Active Server Page (ASP) and COM component, when someone approached me and asked if I knew anything about databases. Without even a pause, I answered a confident yes, most people in IT know "something" about databases, right? Well, it turned out that a big project was starting, and they needed someone to create and manage a database. I acquired a server, installed a relational database management system (RDMBS), and executed CREATE DATABASE dbName; GO. And the rest is history. I like to call that out because these days, most of the data storage architecture already exists when you start the job. You must learn what someone else created. You experience problems but do not know why, because a lot happened before you started.

The new emerging technology called big data is providing a rare opportunity, kind of like the one I had. The opportunity is to build and/or be involved in creating an IT data analytics solution from the beginning. Being the person or the team who builds the framework and foundation of what could become a system that shapes the future of a company is career-altering. The experience is a differentiator that stays with you for the rest of your career, as it has in mine. But it could also be a catastrophe for numerous reasons, such as not being able to scale, being too hard to make changes, and not being reliable.

I must admit that during the early stages I re-created that database numerous times, dropping and re-creating tables, removing primary and foreign keys, changing indexes, reconfiguring triggers, and tuning stored procedures. I knew this was necessary in numerous scenarios because I made all the configurations myself and felt it was easier to start over than it was to troubleshoot further or make a questionable configuration that could have harsh consequences later. As the project matured, it became much more difficult and impactful to make changes; the design could no longer be changed; it was go time, do or die, succeed or fail.

Building the foundation of a project is great, though it can be a bit overwhelming, and at times you might feel you are guessing or simply bulldozing full speed ahead. Those feelings are normal, and some can end up defining you, destroying you, or making you stronger. The data analytics system you design must be able to scale, change, and provide dependable data on request; anything else will ultimately fail. This book can help you learn about scalability, flexibility, and reliability, which are areas that ultimately determine the success of a data analytics solution.

Who This Book Is For

This book is for anyone who wants to learn about Microsoft Azure products and features and, ultimately, to attain the Azure Data Engineer Associate certification. This book is not intended for absolute beginners, although beginners may gain some greater insights into Azure and how to consume and configure its products and services. Gaining the Azure Data Engineer Associate certification means that you can design, develop, implement, monitor, and optimize data solutions using the following:

• Azure Synapse Analytics
• Azure Data Factory (ADF)
• Azure Databricks
• Azure Data Lake Storage (ADLS)
• Azure Storage
• Azure SQL
• Azure Cosmos DB
• Apache Spark
• HDInsight
• Azure Stream Analytics
• Azure Event Hubs / IoT Hub
• Azure Batch

In addition understanding these Azure data products, you need to have some additional knowledge and experience with the following and many other related and dependent Azure products:

• Microsoft Purview
• Azure Virtual Machines
• Azure Functions
• Azure App Service
• Azure DevOps
• Azure Key Vault
• Azure Policy
• Azure Active Directory
• Azure Managed Identity
• Azure Monitor
• Azure Migrate
• Azure Virtual Network

That is a very broad range of topics, and the number of possible scenarios in which to apply them is equally as great. This book will provide insights into each of those topics, plus numerous others, but you are expected to have some basic, preexisting experience with them.

What This Book Covers

This book covers everything you need to know to greatly increase the probability of passing the Azure Data Engineer Associate exam. Of most importance, however, this book will help you design, implement, and support the ingestion, preparation, analysis, and presentation of data. Which of those two scenarios is most important to you? Hopefully both, because that is the goal and purpose of this book. You will learn about many Azure products and features, including Azure Synapse Analytics, Azure SQL, Azure Databricks, Azure Data Factory, Azure security, Azure networking, Azure compute, Azure datastores and storage, Azure messaging services, Azure migration tools, Azure monitoring tools and Azure recovery tools. In addition to learning about what these products and features are and do, you will perform some real-world exercises to implement and use many of them.

How This Book Is Structured

Good design really is everything. Unless you plan before doing it, it is highly probable that the result will not quite measure up to the expectations. In many instances, even with good planning and thought, the result might not measure up or even be successful. There are many priorities and areas to be concerned with when planning a big project. The same is true when you are migrating existing on-premises workloads to Azure or creating new applications and infrastructure directly on Azure. In both scenarios, security, networking, compute, and data storage all come into focus. The chapters in this book are provided in the order of priority. When planning your migration or deployments, make sure to include each of the phases. The order in which those IT components are analyzed, designed, and implemented is important and is the reason the book is constructed in this way.

• Part I: Azure Data Engineer Certification and Azure Products
• Part II: Design and Implement Data Storage
• Part III: Develop Data Processing
• Part IV: Secure, Monitor, and Optimize Data Storage and Data Processing

Security is by far the most important point of concentration. The network needs to be secured before you place your workloads into it. Then your data, compliance and governance, messaging concepts, development concepts, initial application deployment, and updates cannot be ignored or missed. Once your application is deployed, its lifecycle is really just the beginning. Monitoring it and having a failover and disaster recovery plan designed and tested are necessities for production IT solutions.

Following the design pattern laid out by the chapter flow will help you become a great Azure Data Engineer. Note that when you take the Data Engineering on Microsoft Azure exam, you sign a nondisclosure agreement (NDA) stating that you will not discuss the questions or any of the content of the exam. That is very important so that the integrity and value of the credential you gain when passing the exam maintains its prestige. This book will help you learn the skills and gain the experience an Azure Data Engineer should have. By learning and exercising the techniques in this book, your probability of passing the exam will be greatly increased. The point is, the book is geared towards building your experience and skill set on the Azure platform, which will then enable you to gain the certification.

Chapter 1: Gaining the Azure Data Engineer Associate Certification Chapter 1 provides an overview of what it takes to attain the Azure Data Engineer Associate certification. It begins with a quick overview of all Microsoft data-related certifications, such as the Azure Data Scientist and Azure Data Analytics certifications. This way you get good insights into the specific expectations regarding the Azure Data Engineer Associate certification. Comparing different certifications helps you gain a more precise set of Azure Data Engineer skills. You might also consider taking the AZ-900 exam, which is an introduction to Azure certification, and/or the DP-900 exam, which is an introduction to the Azure data service products. Both certifications will benefit your learning progressions in this field, but they are optional and not required for the DP-203 exam. You will also find some tips on how to pass the DP-203 exam, followed by an introduction to many Azure products and features. Although the DP-203 exam focuses on data services such as Azure Synapse Analytics, Azure Data Factory, and Azure Databricks, you must have some knowledge of other Azure products, including Azure Active Directory, Azure Key Vault, Azure Managed Identity, and Azure Stream Analytics, to name a few.

Chapter 2: CREATE DATABASE dbName; GO The first few chapters of a book are usually meant to get the reader into the right mindset so that the content in the succeeding chapters can be digested more easily. In this case, however, buckle your seatbelt and get ready to hit the ground running, because there is no time to waste. Chapter 2 briefly introduces the world of data but then dives right into some rather sophisticated data structures, types, and concepts an Azure Data Engineer Associate must know. To some extent, the discussion on those topics should be a refresher for an experienced data engineer who intends on taking the Azure Data...