Stream Analytics with Microsoft Azure
Real-time data processing for quick insights using Azure Stream Analytics
1st Edition
Published on 3. February 2025
322 pages
978-1-78839-062-0 (ISBN)
Description
No detailed description available for "Stream Analytics with Microsoft Azure".
More details
Persons
Murphy Ryan :
Ryan Murphy, one of the key Co-authors of Stream Analytics with Microsoft Azure is a Solution Architect living in Saint Louis, Missouri. He has been building and innovating with data for nearly twenty years, including extensive work in the gaming and agriculture industries. Currently, Ryan is helping some of the worlds largest organizations modernize their business with data solutions powered by the Microsoft Azure cloud.Singh Manpreet :
Manpreet Singh is a consultant and author with extensive expertise in architecture, design, and implementation of business intelligence and Big Data analytics solutions. He is passionate about enabling businesses to derive valuable insights from their data. Manpreet has been working on Microsoft technologies for more than 10 years, with a strong focus on Microsoft Business Intelligence Stack, SharePoint BI, and Microsofts Big Data Analytics Platforms (Analytics Platform System and HDInsight). He also specializes in Mobile Business Intelligence solution development and has helped businesses deliver a consolidated view of their data to their mobile workforces. Manpreet has coauthored books and technical articles on Microsoft technologies, focusing on the development of data analytics and visualization solutions with the Microsoft BI Stack and SharePoint. He holds a degree in computer science and engineering from Punjab University, India.
Ryan Murphy, one of the key Co-authors of Stream Analytics with Microsoft Azure is a Solution Architect living in Saint Louis, Missouri. He has been building and innovating with data for nearly twenty years, including extensive work in the gaming and agriculture industries. Currently, Ryan is helping some of the worlds largest organizations modernize their business with data solutions powered by the Microsoft Azure cloud.Singh Manpreet :
Manpreet Singh is a consultant and author with extensive expertise in architecture, design, and implementation of business intelligence and Big Data analytics solutions. He is passionate about enabling businesses to derive valuable insights from their data. Manpreet has been working on Microsoft technologies for more than 10 years, with a strong focus on Microsoft Business Intelligence Stack, SharePoint BI, and Microsofts Big Data Analytics Platforms (Analytics Platform System and HDInsight). He also specializes in Mobile Business Intelligence solution development and has helped businesses deliver a consolidated view of their data to their mobile workforces. Manpreet has coauthored books and technical articles on Microsoft technologies, focusing on the development of data analytics and visualization solutions with the Microsoft BI Stack and SharePoint. He holds a degree in computer science and engineering from Punjab University, India.
Content
- Cover
- Copyright
- Credits
- About the Authors
- About the Reviewers
- www.PacktPub.com
- Customer Feedback
- Table of Contents
- Preface
- Chapter 1: Introducing Stream Processing and Real-Time Insights
- Understanding stream processing
- Understanding queues, Pub/Sub, and events
- Queues
- Publish and Subscribe model
- Real-world implementations of the Publish/Subscribe model
- Azure implementation of queues and Publish/Subscribe models
- What is an event?
- Event streaming
- Event correlation
- Azure implementation of event processing
- Architectural components of Event Hubs
- Simple event processing
- Event stream processing
- Complex event processing
- Summary
- Chapter 2: Introducing Azure Stream Analytics and Key Advantages
- Services offered by Microsoft
- Introduction to Azure Stream Analytics
- Configuration of Azure Stream Analytics
- Key advantages of Azure Stream Analytics
- Security
- Programmer productivity
- Declarative SQL constructs
- Built-in temporal semantics
- Lowest total cost of ownership
- Mission-critical and enterprise-less scalability and availability
- Global compliance
- Microsoft Cortana Intelligence suite integration
- Azure IoT integration
- Summary
- Chapter 3: Designing Real-Time Streaming Pipelines
- Differencing stream processing and batch processing
- Logical flow of processing
- Out of order and late arrival of data
- Session grouping and windowing challenges
- Message consistency
- Fault tolerance, recovery, and storage
- Source
- Communication and collection
- Ingest, queue, and transform
- Hot path
- Cold path
- Data retention
- Presentation and action
- Canonical Azure architecture
- Summary
- Chapter 4: Developing Real-Time Event Processing with Azure Streaming
- Stream Analytics tools for Visual Studio
- Prerequisites for the installation of Stream Analytics tools
- Development of a Stream Analytics job using Visual Studio
- Defining a Stream Analytics query for Vehicle Telemetry job analysis using Stream Analytics tools
- Query to define Vehicle Telemetry (Connected Car) engine health status and pollution index over cities
- Testing Stream Analytics queries locally or in the cloud
- Stream Analytics job configuration parameter settings in Visual Studio
- Implementation of an Azure Stream Analytics job using the Azure portal
- Provisioning for an Azure Stream Analytics job using the Azure Resource Manager template
- Azure ARM Template - Infrastructure as code
- Getting started with provisioning Azure Stream Analytics job using the ARM template
- Deployment and validation of the Stream Analytics ARM template to Azure Resource Group
- Configuration of the Azure Streaming job with different input data sources and output data sinks
- Data input types-data stream and reference data
- Data Stream inputs
- Reference data
- Job topology output data sinks of Stream Analytics
- Summary
- Chapter 5: Building Using Stream Analytics Query Language
- Built-in functions
- Scalar functions
- Aggregate and analytic functions
- Array functions
- Other functions
- Data types and formats
- Complex types
- Query language elements
- Windowing
- Tumbling windows
- Hopping windows
- Sliding windows
- Time management and event delivery guarantees
- Summary
- Chapter 6: How to achieve Seamless Scalability with Automation
- Understanding parts of a Stream Analytics job definition (input, output, reference data, and job)
- Deployment of Azure Stream Analytics using ARM template
- Configuring input
- Configuring output
- Building the sample test code
- How to scale queries using Streaming units and partitions
- Application and Arrival Time
- Partitions
- Input source
- Output source
- Embarrassingly parallel jobs and Not embarrassingly parallel jobs
- Sample use case
- Configuring SU using Azure portal
- Out of order and late-arriving events
- Summary
- Chapter 7: Integration of Microsoft Business Intelligence and Big Data
- What is Big Data Lambda Architecture?
- Concepts of batch processing and stream processing in data analytics
- Specifications for slow/cold path of data - batch data processing
- Moving to the streaming-based data solution pattern
- Evolution of Kappa Architecture and benefits
- Comparison between Azure Stream Analytics and Azure HDInsight Storm
- Designing data processing pipeline of an interactive visual dashboard through Stream Analytics and Power BI
- Integrating Power BI as an output job connector for Stream Analytics
- Summary
- Chapter 8: Designing and Managing Stream Analytics Jobs
- Reference data streams with Azure Stream Analytics
- Configuration of Reference data for Azure Stream Analytics jobs
- Integrating a reference data stream as job topology input for an Azure Stream Analytics job
- Stream Analytics query configuration for Reference Data join
- Refresh schedule of a reference data stream
- Configuration of output data sinks for Azure Stream Analytics with Azure Data Lake Store
- Configuring Azure Data Lake Store as an output data sink of Stream Analytics
- Configuring Azure Data Lake Store as an output sink of Stream Analytics jobs
- Configuring Azure Cosmos DB as an output data sink for Azure Stream Analytics
- Features of Azure Cosmos DB for configuring output sinks of Azure Stream Analytics
- Configuring Azure Cosmos DB integrated with Azure Stream Analytics as an output sink
- Stream Analytics job output to Azure Function Apps as Serverless Architecture
- Provisioning steps to an Azure Function
- Configuring an Azure function as a serverless architecture model integrated with Stream Analytics job output
- Summary
- Chapter 9: Optimizing Intelligence in Azure Streaming
- Integration of JavaScript user-defined functions using Azure Stream Analytics
- Adding JavaScript UDF with a Stream Analytics job
- Stream Analytics and JavaScript data type conversions
- Integrating intelligent Azure machine learning algorithms with Stream Analytics function
- Data pipeline Streaming application building concepts using Azure .NET Management SDK
- Implementation steps of Azure Stream Analytics jobs using .NET management SDK
- Summary
- Chapter 10: Understanding Stream Analytics Job Monitoring
- Troubleshooting with job metrics
- Visual monitoring of job diagram
- Logging of diagnostics logs
- Enabling diagnostics logs
- Exploring the logs sent to the storage account
- Configuring job alerts
- Viewing resource health information with Azure resource health
- Exploring different monitoring experiences
- Building a monitoring dashboard
- Summary
- Chapter 11: Use Cases for Real-World Data Streaming Architectures
- Solution architecture design and Proof-of-Concept implementation of social media sentiment analytics using Twitter and a sentiment analytics dashboard
- Definition of sentiment analytics
- Prerequisites required for the implementation of Twitter sentiment analytics PoC
- Steps for implementation of Twitter sentiment analytics
- Remote monitoring analytics using Azure IoT Suite
- Provisioning of remote device monitoring analytics using Azure IoT Suite
- Implementation of a connected factory use case using Azure IoT Suite
- Connected factory solution with Azure IoT Suite
- Real-world telecom fraud detection analytics using Azure Stream Analytics and Cortana Intelligence Gallery with interactive visuals from Microsoft Power BI
- Implementation steps of fraud detection analytics using Azure Stream Analytics
- Steps for building the fraud detection analytics solution
- Summary
- Index
