Modern Network Observability
Description
Key Features
Craft a dynamic observability stack with real-world, practical applications
Build intuitive dashboards and alerts by collecting and normalizing diverse network data
Leverage observability data to strengthen automation strategies for network operations
Purchase of the print or Kindle book includes a free PDF eBook
Book DescriptionAs modern IT services and software architectures such as microservices rely increasingly on network performance, the relevance of networks has never been greater. Network observability has emerged as a critical evolution of traditional monitoring, providing the deep visibility needed to manage today's complex, dynamic environments. In Modern Network Observability, authors David Flores, Christian Adell, and Josh VanDeraa share their extensive experience to guide you through building and deploying a flexible observability stack using open-source tools. This book begins by addressing the limitations of monolithic monitoring solutions, showing you how to transform them into a composable, flexible observability stack. Through practical implementations, you'll learn how to collect, normalize, and analyze network data from diverse sources, build intuitive dashboards, and set up actionable alerts that help you stay ahead of potential issues. Later, you'll cover advanced topics, such as integrating observability data into your network automation strategy, ensuring your network operations align with business objectives. By the end of this book, you'll be able to proactively manage your network, minimize downtime, and ensure resilient, efficient, and future-proof operations.What you will learn
Collect and normalize data from various sources using Telegraf and Logstash
Enrich operational data with crucial context from a Source of Truth such as Nautobot
Visualize data and create insightful dashboards with Grafana
Automate alerts and responses for your network operations strategy using Prefect
Understand when to build or buy an observability stack, with tips and best practices
Explore practical machine learning techniques to enhance observability data value
Who this book is forThis book is for all network engineering roles such as network analysts, administrators, architects, security personnel, support staff, and managers working in both on-premises and cloud environments who are tasked with implementing or using network monitoring solutions. Basic programming knowledge in Python and Go, familiarity with networking concepts, and a fundamental understanding of Docker containers for lab scenarios will be required.
All prices
More details
Other editions
Additional editions
Persons
David Flores is passionate about solving complex problems in network infrastructure, software architectures, automation, and observability. With experience with service providers, cloud providers, and system integrators, David has gained expertise in managing, automating, and building observability stacks for network infrastructure. Currently at CoreWeave, he focuses on enhancing automation and observability. David has also contributed to open source projects such as gns3fy, and actively shares his knowledge through blogs, workshops, and technical events. David is always curious and eager to keep himself updated and open to new ideas in the field.Adell Christian :
Christian Adell is a principal architect at Network to Code He is focused on building network automation solutions for diverse use cases, with great emphasis on open source software. He is passionate about learning and helping others to grow, but also has more hobbies than hours in the day, so working remotely from Barcelona gives him the time and the space to achieve his dreams. Christian is a co-author of O'Reilly's Network Programmability and Automation book and a co-author of Network Automation with Nautobot by Packt . Also in relation to sharing knowledge, he is the organizer of the NetBCN community in Barcelona and has been collaborating with several universities for almost 20 years.VanDeraa Josh :
Josh VanDeraa is a network engineer and automation leader. Currently, he is a services director at Network to Code, driving value from network automation solutions. Josh has experience in automation and networking across retail, transportation, and managed services. In his free time, he enjoys being with his family or the Minnesota seasons. Josh co-authored Network Automation with Nautobot and self-published Open Source Network Management.
Content
Introduction to Monitoring and Observability
Role of Monitoring and Observability in Network Infrastructure
Data's Role in Network Observability
Observability Stack Architecture
Data Collectors
Data Distribution and Processing
Data Storage Solutions for Network Observability
Visualization - Bringing Network Observability to Life
Alerting - Network Monitoring and Incident Management
Real-World Observability Architectures
Applications of Your Observability Data - Driving Business Success
Automation Powered by Observability Data - Streamlining Network Operations
Leveraging Artificial Intelligence for Enhanced Network Observability
Appendix A
1
Introduction to Monitoring and Observability
Since the early days of computer networks, we have needed to detect failures on the different network components (e.g., hardware interface issues, cable cuts, or web service down) to determine outages that require corrective actions. This field has been known as network monitoring.
Interestingly, the last decade has witnessed numerous innovations in the field, especially related to new tools and practices around the DevOps culture. This culture emphasizes merging development and operations responsibilities requiring a better understanding of the operational state. Moreover, there has been a significant adoption of network automation. This advancement drives network operations, transforming monitoring from a passive component to an enabler of closed-loop processes. These changes have been the main drivers behind the evolution from network monitoring to network observability, and this book wants to help you understand and apply it to improve your network operations.
Note
Network observability is a broader topic, especially since the rise of running network applications directly in the host with technologies such as extended Berkeley Packet Filter (eBPF) and Data Plane Development Kit (DPDK). This kind of observability is not covered in detail in the book, even though most of the concepts are applicable too.
In this book, you will begin understanding the basics concepts related to network observability, and then, for the majority of it, we will explain how to build a modern network observability stack, with a practical, but not limited, emphasis on the Telegraf (https://github.com/influxdata/telegraf)/Prometheus (https://github.com/prometheus/prometheus)/Grafana (https://github.com/grafana/grafana) (TPG) stack (details about how to spin up a development environment are in Appendix A). Finally, you will learn how to solve real network operations challenges using the flexible observability stack presented.
In this first chapter, we will cover the following topics:
- Defining network observability
- Describing network monitoring evolution
- Exposing the key aspects of network observability
Defining network observability
Let's go straight to the point: what is network observability about?
To answer this, it's convenient to understand first what network monitoring is because network observability supersedes it. Network monitoring is part of the wide IT operations monitoring focused on the network infrastructure.
Even though you are likely used to the network monitoring term, there is no academic definition of it, and everyone understands it slightly differently. We define network monitoring as measuring the performance and availability of the network infrastructure.
Related to this goal, you may be familiar with some of the technologies that have provided information about the operational state of the network:
- Simple Network Management Protocol (SNMP) polls and traps
- Internet Control Message Protocol (ICMP) requests (e.g., ping)
- Flow analysis (e.g., NetFlow)
- Packet capture (e.g., tcpdump)
- Logs (e.g., Syslog)
These technologies make up network monitoring, which provides support for diagnostics and service monitoring, with state visualization and alert generation. Network operation teams leverage network monitoring to detect when something is wrong in the network, but this is not enough anymore.
Nowadays, IT operations have raised the bar, and the focus is not only on the infrastructure status but on translating it to the business level. Therefore, observability is about the end user's experience, and this encompasses many layers, from infrastructure to applications.
This convergence of responsibilities materialized in the DevOps culture (i.e., bringing together Development and Operations) that coordinates all the IT efforts around the same business outcome. One basic practice is to consolidate different monitoring systems to enable data correlation. The DevOps movement has broken long-time silos in IT departments, and this new collaboration has produced a lot of innovations, which we will explore in this book.
Moreover, it has transformed the reactive approach of traditional monitoring into a proactive one that helps answer handling issues before impacting the services. Ironically, this leads to simpler (but more effective) systems, capable of getting the data to provide the insights that help solve these issues. This is what IT observability is about, helping to identify the unknown unknowns and having a holistic view.
Within this observability realm, network observability encompasses all the technological trends that support the overall IT observability in the network realm.
In networking, this trend toward adopting network observability has been translated to more flexibility in different aspects:
- Interoperable specialized solutions (e.g., open source solutions provide more flexibility)
- More efficient data retrieval methods (e.g., network streaming telemetry)
- More scalable and advanced data processing (e.g., artificial intelligence)
- Richer context and analysis via data integrations (e.g., source of truth integration)
Note
That being said, we will use both terms (i.e., monitoring and observability) interchangeably in this book, with the same meaning.
This is what this book is about. We want you to understand how to evolve from traditional network monitoring systems to the new network observability approach, tightly connected with the DevOps culture, and how it connects with the other big revolution in network operations: network automation.
Network monitoring evolution
As already mentioned, modern network observability has evolved from network monitoring, a practice that has been in place for several decades. Before delving into the new approach it introduces, it's important to review what has been effective so far and to understand the trends and requirements that have driven its transformation.
What has worked so far
Networks have been monitored to understand their status since the beginning. ARPANET (which stands for Advanced Research Projects Agency Network), the first packet-switched network started in 1966, had the Interface Message Processor (IMP) protocol, which provided a few monitoring features. Fast-forwarding some years to the rise of TCP/IP networks, in 1988, the SNMP was defined by the IETF (its last version is SNMPv3) to address this need.
SNMP provides a mechanism to manage networks, but it has been mostly used to monitor networks, and not to manage configuration changes (which have been mostly done via CLIs, until the rise of newer management interfaces). The main characteristics of SNMP can be summarized in a few aspects:
- The UDP transport protocol is stateless, which is useful for state and status polling
- Management information bases (MIBs) provide structured data to access specific content
- Massive adoption in all network devices, supporting standard and proprietary MIBs
However, not all that glitters is gold, and SNMP has some limitations such as the performance to retrieve large amounts of data and limited coverage for push mechanisms (i.e., SNMP traps).
Note
This book doesn't cover SNMP in detail (there are many books dedicated to the topic). We will reference it as one of the available methods to retrieve operational data within a holistic network observability strategy in Chapter 3.
Similarly to SNMP, event logs using Syslog have been widely used, not only for network monitoring but also for applications. Logs are generated when a specific event is seen by the device, and it brings together several pieces of information such as the generation time, the source, the level, and some meaningful message related to the event. This grouping of data is what we refer to as multidomain data. This contrasts with the simple SNMP metrics (integers or strings).
And also, pretty common in network analysis are the flow exporters mechanisms such as NetFlow, sFlow, and IPFIX. With some small differences between them, they represent the basic information to define what a packet flow is about, including the source and...
