ACI Advanced Monitoring and Troubleshooting
Published on 5. February 2021
Book
Paperback/Softback
944 pages
978-1-58714-528-5 (ISBN)
Description
Advanced real-world Cisco Application Centric Infrastructure (ACI) monitoring and troubleshooting
This expert guide and reference will help you confidently deploy, support, monitor, and troubleshoot ACI fabrics and components. It is also designed to help you prepare for your Cisco DCACIA (300-630) exam, earning Cisco Certified Specialist-ACI Advanced Implementation certification and credit toward CCNP Data Center certification if you choose.
Authored by three leading Cisco ACI experts, it combines a solid conceptual foundation, in-depth technical knowledge, and practical techniques. It also contains proven features to help exam candidates prepare, including review questions in most chapters, and Key Topic icons highlighting concepts covered on the exam.
The authors thoroughly introduce ACI functions, components, policies, command-line interfaces, connectivity, fabric design, virtualization and service integration, automation, orchestration, and more. Next, they introduce best practices for monitoring and management, including the use of faults, health scores, tools, the REST API, in-band and out-of-band management techniques, and monitoring protocols. Proven configurations are provided, with steps for verification. Finally, they present advanced forwarding and troubleshooting techniques for maximizing ACI performance and value.
ACI Advanced Monitoring and Troubleshooting is an indispensable resource for every data center architect, engineer, developer, network or virtualization administrator, and operations team member working in ACI environments.
Understand Cisco ACI core functions, components, and protocols
Apply the ACI Policy-Based Object Model to develop overall application frameworks
Use command-line interfaces to manage and monitor Cisco ACI systems
Master proven options for ACI physical and logical fabric design
Establish connectivity for compute, storage, and service devices, switches, and routers
Gain visibility into virtualization layers through VMM, and integrate hypervisors from multiple vendors
Seamlessly integrate Layer 4 to Layer 7 services such as load balancing and firewalling
Automate and orchestrate for fast deployment with the REST API, scripting, and Ansible
Minimize downtime and maximize ROI through more effective monitoring and configuration
Thoroughly master concepts and techniques for advanced ACI and VXLAN forwarding
Build deep practical expertise for quickly troubleshooting critical events
Gain quick visibility into traffic flows and streamline problem isolation with the ACI Visibility & Troubleshooting Tool
Walk through multiple real-world troubleshooting scenarios step-by-step
Forewords written by Yusuf Bhaiji, Director of Certifications, Cisco Systems; and Ronak Desai, VP of Engineering for the Data Center Networking Business Unit, Cisco Systems.
This book is part of the Networking Technology Series from Cisco Press, which offers networking professionals valuable information for constructing efficient networks, understanding new technologies, and building successful careers.
This expert guide and reference will help you confidently deploy, support, monitor, and troubleshoot ACI fabrics and components. It is also designed to help you prepare for your Cisco DCACIA (300-630) exam, earning Cisco Certified Specialist-ACI Advanced Implementation certification and credit toward CCNP Data Center certification if you choose.
Authored by three leading Cisco ACI experts, it combines a solid conceptual foundation, in-depth technical knowledge, and practical techniques. It also contains proven features to help exam candidates prepare, including review questions in most chapters, and Key Topic icons highlighting concepts covered on the exam.
The authors thoroughly introduce ACI functions, components, policies, command-line interfaces, connectivity, fabric design, virtualization and service integration, automation, orchestration, and more. Next, they introduce best practices for monitoring and management, including the use of faults, health scores, tools, the REST API, in-band and out-of-band management techniques, and monitoring protocols. Proven configurations are provided, with steps for verification. Finally, they present advanced forwarding and troubleshooting techniques for maximizing ACI performance and value.
ACI Advanced Monitoring and Troubleshooting is an indispensable resource for every data center architect, engineer, developer, network or virtualization administrator, and operations team member working in ACI environments.
Understand Cisco ACI core functions, components, and protocols
Apply the ACI Policy-Based Object Model to develop overall application frameworks
Use command-line interfaces to manage and monitor Cisco ACI systems
Master proven options for ACI physical and logical fabric design
Establish connectivity for compute, storage, and service devices, switches, and routers
Gain visibility into virtualization layers through VMM, and integrate hypervisors from multiple vendors
Seamlessly integrate Layer 4 to Layer 7 services such as load balancing and firewalling
Automate and orchestrate for fast deployment with the REST API, scripting, and Ansible
Minimize downtime and maximize ROI through more effective monitoring and configuration
Thoroughly master concepts and techniques for advanced ACI and VXLAN forwarding
Build deep practical expertise for quickly troubleshooting critical events
Gain quick visibility into traffic flows and streamline problem isolation with the ACI Visibility & Troubleshooting Tool
Walk through multiple real-world troubleshooting scenarios step-by-step
Forewords written by Yusuf Bhaiji, Director of Certifications, Cisco Systems; and Ronak Desai, VP of Engineering for the Data Center Networking Business Unit, Cisco Systems.
This book is part of the Networking Technology Series from Cisco Press, which offers networking professionals valuable information for constructing efficient networks, understanding new technologies, and building successful careers.
More details
Series
Language
English
Place of publication
Indianapolis
United States
Publishing group
Pearson Education (US)
Target group
Professional and scholarly
Product notice
Paperback (trade)
Unsewn / adhesive bound
Dimensions
Height: 235 mm
Width: 195 mm
Thickness: 51 mm
Weight
1759 gr
ISBN-13
978-1-58714-528-5 (9781587145285)
Copyright in bibliographic data and cover images is held by Nielsen Book Services Limited or by the publishers or by their respective licensors: all rights reserved.
Schweitzer Classification
Persons
Sadiq Memon, CCIE No. 47508, is a Lead Solutions Integration Architect (Automotive) with Cisco Customer Experience (CX). He has over 30 years of diversified experience in information technology with specialization and expertise in data center and enterprise networking. Sadiq joined Cisco in 2007, and as a Cisco veteran of over 13 years, he has worked with various large enterprise customers, including automotive, financials, manufacturing, and government in designing, implementing, and supporting end-to-end architectures and solutions. Sadiq was part of the Cisco Advanced Services Tiger Team during the early ACI incubation period. He has published a series of short videos covering ACI configuration on YouTube and has presented ACI/Cloud-related topics at Cisco Live! Sadiq was the technical editor for the Cisco Press book Deploying ACI and possesses multiple IT industry certifications from leading companies such as Cisco (CCIE, CCNA), VMware (VCP-DCV), Microsoft, and Citrix. Sadiq holds a bachelor's degree in computer systems engineering from NED University of Engineering & Technology, Karachi, Pakistan.
Joseph Ristaino, CCIE No. 41799, is a Technical Leader with the ACI Escalation Team in RTP, North Carolina. He joined Cisco in 2011 after graduating from Wentworth Institute of Technology with a bachelor's degree in computer networking. Joseph started with Cisco on the Server Virtualization TAC team, specializing in UCS and virtualization technologies. He has in-depth knowledge of compute/networking technologies and has been supporting customers for over eight years as they implement and manage data center deployments around the globe. Joseph now works closely with the ACI Technical Support teams to provide assistance on critical customer issues that go unsolved and has been working on ACI since its inception in 2014. Joseph lives with his wife in Durham, North Carolina.
Carlo Schmidt, CCIE No. 41842, is a Data Center Solutions Architect. He works with global enterprises, designing their next-generation data centers. Carlo started at Cisco in 2011, on the Data Center Switching TAC team. In that role, he focused on Nexus platforms and technologies such as FCoE, fabric path, and OTV. In 2016, he migrated to the ACI TAC team, where he specialized in customer problem resolution as well as improving product usability. In 2019 Carlo decided to take his knowledge and lessons learned from his eight years in Cisco TAC to a presales role as a Solutions Architect. Carlo is based out of Research Triangle Park, North Carolina.
Joseph Ristaino, CCIE No. 41799, is a Technical Leader with the ACI Escalation Team in RTP, North Carolina. He joined Cisco in 2011 after graduating from Wentworth Institute of Technology with a bachelor's degree in computer networking. Joseph started with Cisco on the Server Virtualization TAC team, specializing in UCS and virtualization technologies. He has in-depth knowledge of compute/networking technologies and has been supporting customers for over eight years as they implement and manage data center deployments around the globe. Joseph now works closely with the ACI Technical Support teams to provide assistance on critical customer issues that go unsolved and has been working on ACI since its inception in 2014. Joseph lives with his wife in Durham, North Carolina.
Carlo Schmidt, CCIE No. 41842, is a Data Center Solutions Architect. He works with global enterprises, designing their next-generation data centers. Carlo started at Cisco in 2011, on the Data Center Switching TAC team. In that role, he focused on Nexus platforms and technologies such as FCoE, fabric path, and OTV. In 2016, he migrated to the ACI TAC team, where he specialized in customer problem resolution as well as improving product usability. In 2019 Carlo decided to take his knowledge and lessons learned from his eight years in Cisco TAC to a presales role as a Solutions Architect. Carlo is based out of Research Triangle Park, North Carolina.
Content
Foreword by Yusuf Bhaiji xxviii
Foreword by Ronak Desai xxix
Introduction xxx
PART I: INTRODUCTION TO ACI
Chapter 1 Fundamental Functions and Components of Cisco ACI 1
ACI Building Blocks 8
Hardware Specifications 8
ACI Key Concepts 14
Control Plane 15
Data Plane 17
VXLAN 17
Tenant 18
VRF 19
Application Profile 20
Endpoint Group 21
Contracts 22
Bridge Domain 24
External Routed or Bridged Network 25
Summary 26
Review Key Topics 26
Review Questions 27
Chapter 2 Introduction to the ACI Policy Model 31
Key Characteristics of the Policy Model 32
Management Information Tree (MIT) 33
Benefits of a Policy Model 37
Logical Constructs 37
Tenant Objects 38
VRF Objects 39
Application Profile Objects 40
Endpoint Group Objects 41
Bridge Domain and Subnet Objects 43
Bridge Domain Options 45
Contract Objects 46
Labels, Filters, and Aliases 48
Contract Inheritance 49
Contract Preferred Groups 49
vzAny 50
Outside Network Objects 51
Physical Construct 52
Access Policies 52
Switch Policies 53
Interface Policies 54
Global Policies 55
Managed Object Relationships and Policy Resolution 57
Tags 58
Default Policies 58
How a Policy Model Helps in Diagnosis 60
Summary 63
Review Key Topics 63
Review Questions 64
Chapter 3 ACI Command-Line Interfaces 67
APIC CLIs 68
NX-OS-Style CLI 68
Bash CLI 74
ACI Fabric Switch CLIs 78
iBash CLI 78
VSH CLI 81
VSH_LC CLI 83
Summary 84
Reference 84
Chapter 4 ACI Fabric Design Options 85
Physical Design 85
Single- Versus Multiple-Fabric Design 87
Multi-Pod 97
Multi-Site 116
Remote Leaf 131
Hardware and Software Support 134
ACI Multi-Pod and Remote Leaf Integration 143
Logical Design 149
Design 1: Container-as-a-Service Using the OpenShift Platform and Calico CNI 149
Design 2: Vendor-Based ERP/SAP Hana Design with ACI 165
Design 3: vBrick Digital Media Engine Design with ACI 175
Summary 180
Review Key Topics 181
Review Questions 181
Chapter 5 End Host and Network Connectivity 185
End Host Connectivity 185
VLAN Pool 186
Domain 186
Attachable Access Entity Profiles (AAEPs) 186
Switch Policies 187
Interface Policies 188
Virtual Port Channel (VPC) 191
Port Channel 197
Access Port 201
Best Practices in Configuring Access Policies 206
Compute and Storage Connectivity 207
L4/L7 Service Device Connectivity 210
Network Connectivity 213
Connecting an External Bridge Network 213
Connecting an External Routed Network 218
Diagnosing Connectivity Problems 242
Summary 245
Review Questions 245
Chapter 6 VMM Integration 249
Virtual Machine Manager (VMM) 249
VMM Domain Policy Model 250
VMM Domain Components 250
VMM Domains 250
VMM Domain VLAN Pool Association 252
VMware Integration 257
Prerequisites for VMM Integration with AVS or VDS 257
Guidelines and Limitations for VMM Integration with AVS or VDS 257
ACI VMM Integration Workflow 258
Publishing EPGs to a VMM Domain 258
Connecting Virtual Machines to the Endpoint Group Port Groups on vCenter 259
Verifying VMM Integration with the AVS or VDS 259
Microsoft SCVMM Integration 260
Mapping ACI and SCVMM Constructs 261
Mapping Multiple SCVMMs to an APIC 262
Verifying That the OpFlex Certificate Is Deployed for a Connection from the SCVMM to the APIC 262
Verifying VMM Deployment from the APIC to the SCVMM 263
OpenStack Integration 263
Extending OpFlex to the Compute Node 264
ACI with OpenStack Physical Architecture 264
OpFlex Software Architecture 265
OpenStack Logical Topology 265
Mapping OpenStack and ACI Constructs 266
Kubernetes Integration 272
Planning for Kubernetes Integration 272
Prerequisites for Integrating Kubernetes with Cisco ACI 273
Provisioning Cisco ACI to Work with Kubernetes 274
Preparing the Kubernetes Nodes 277
Installing Kubernetes and Cisco ACI Containers 279
Verifying the Kubernetes Integration 280
OpenShift Integration 281
Planning for OpenShift Integration 282
Prerequisites for Integrating OpenShift with Cisco ACI 283
Provisioning Cisco ACI to Work with OpenShift 284
Preparing the OpenShift Nodes 287
Installing OpenShift and Cisco ACI Containers 290
Updating the OpenShift Router to Use the ACI Fabric 291
Verifying the OpenShift Integration 291
VMM Integration with ACI at Multiple Locations 292
Multi-Site 292
Remote Leaf 295
Summary 298
Chapter 7 L4/L7 Service Integration 299
Service Insertion 299
The Service Graph 300
Managed Mode Versus Un-Managed Mode 301
L4-L7 Integration Use Cases 302
How Contracts Work in ACI 303
The Shadow EPG 306
Configuring the Service Graph 307
Service Graph Design and Deployment Options 312
Policy-Based Redirect (PBR) 322
PBR Design Considerations 323
PBR Design Scenarios 324
Configuring the PBR Service Graph 325
Service Node Health Check 326
Common Issues in the PBR Service Graph 328
L4/L7 Service Integration in Multi-Pod and Multi-Site 332
Multi-Pod 332
Multi-Site 338
Review Questions 342
Chapter 8 Automation and Orchestration 343
The Difference Between Automation and Orchestration 343
Benefits of Automation and Orchestration 344
REST API 349
Automating Tasks Using the Native REST API: JSON and XML 351
API Inspector 351
Object (Save As) 353
Visore (Object Store Browser) 355
MOQuery 357
Automation Use Cases 364
Automating Tasks Using Ansible 372
Ansible Support in ACI 375
Installing Ansible and Ensuring a Secure Connection 378
APIC Authentication in Ansible 382
Automation Use Cases 384
Orchestration Through UCS Director 392
Management Through Cisco UCS Director 392
Automation and Orchestration with Cisco UCS Director 393
Automation Use Cases 395
Summary 402
Review Questions 402
PART II: MONITORING AND MANAGEMENT BEST PRACTICES
Chapter 9 Monitoring ACI Fabric 405
Importance of Monitoring 405
Faults and Health Scores 407
Faults 407
Health Scores 411
ACI Internal Monitoring Tools 415
SNMP 415
Syslog 420
NetFlow 426
ACI External Monitoring Tools 430
Network Insights 430
Network Assurance Engine 437
Tetration 453
Monitoring Through the REST API 473
Monitoring an APIC 475
Monitoring Leafs and Spines 482
Monitoring Applications 499
Summary 505
Review Questions 506
Chapter 10 Network Management and Monitoring Configuration 509
Out-of-Band Management 509
Creating Static Management Addresses 510
Creating the Management Contract 510
Choosing the Node Management EPG 513
Creating an External Management Entity EPG 513
Verifying the OOB Management Configuration 515
In-Band Management 517
Creating a Management Contract 517
Creating Leaf Interface Access Policies for APIC INB Management 518
Creating Access Policies for the Border Leaf(s) Connected to L3Out 520
Creating INB Management External Routed Networks (L3Out) 522
Creating External Management EPGs 524
Creating an INB BD with a Subnet 527
Configuring the Node Management EPG 529
Creating Static Management Addresses 530
Verifying the INB Management Configuration 530
AAA 533
Configuring Cisco Secure ACS 533
Configuring Cisco ISE 542
Configuring AAA in ACI 547
Recovering with the Local Fallback User 550
Verifying the AAA Configuration 550
Syslog 551
Verifying the Syslog Configuration and Functionality 555
SNMP 556
Verifying the SNMP Configuration and Functionality 562
SPAN 566
Access SPAN 567
Fabric SPAN 571
Tenant SPAN 572
Ensuring Visibility and Troubleshooting SPAN 575
Verifying the SPAN Configuration and Functionality 576
NetFlow 577
NetFlow with Access Policies 580
NetFlow with Tenant Policies 582
Verifying the NetFlow Configuration and Functionality 585
Summary 587
PART III: ADVANCED FORWARDING AND TROUBLESHOOTING TECHNIQUES
Chapter 11 ACI Topology 589
Physical Topology 589
APIC Initial Setup 593
Fabric Access Policies 595
Switch Profiles, Switch Policies, and Interface Profiles 595
Interface Policies and Policy Groups 596
Pools, Domains, and AAEPs 597
VMM Domain Configuration 601
VMM Topology 601
Hardware and Software Specifications 603
Logical Layout of EPGs, BDs, VRF Instances, and Contracts 605
L3Out Logical Layout 606
Summary 608
Review Key Topics 608
References 609
Chapter 12 Bits and Bytes of ACI Forwarding 611
Limitations of Traditional Networks and the Evolution of Overlay Networks 611
High-Level VXLAN Overview 613
IS-IS, TEP Addressing, and the ACI Underlay 615
IS-IS and TEP Addressing 615
FTags and the MDT 618
Endpoint Learning in ACI 626
Endpoint Learning in a Layer 2-Only Bridge Domain 627
Endpoint Learning in a Layer 3-Enabled Bridge Domain 635
Fabric Glean 640
Remote Endpoint Learning 641
Endpoint Mobility 645
Anycast Gateway 647
Virtual Port Channels in ACI 649
Routing in ACI 651
Static or Dynamic Routes 651
Learning External Routes in the ACI Fabric 656
Transit Routing 659
Policy Enforcement 661
Shared Services 664
L3Out Flags 668
Quality of Service (QoS) in ACI 669
Externally Set DSCP and CoS Markings 671
CoS Preservation in ACI 672
Multi-Pod 674
Multi-Site 680
Remote Leaf 684
Forwarding Scenarios 686
ARP Flooding 686
Layer 2 Known Unicast 688
ARP Optimization 690
Layer 2 Unknown Unicast Proxy 690
L3 Policy Enforcement When Going to L3Out 693
L3 Policy Enforcement for External Traffic Coming into the Fabric 695
Route Leaking/Shared Services 695
Consumer to Provider 695
Provider to Consumer 698
Multi-Pod Forwarding Examples 698
ARP Flooding 700
Layer 3 Proxy Flow 700
Multi-Site Forwarding Examples 703
ARP Flooding 703
Layer 3 Proxy Flow 705
Remote Leaf 707
ARP Flooding 707
Layer 3 Proxy Flow 710
Summary 713
Review Key Topics 713
References 714
Review Questions 714
Chapter 13 Troubleshooting Techniques 717
General Troubleshooting 717
Faults, Events, and Audits 718
moquery 722
iCurl 724
Visore 726
Infrastructure Troubleshooting 727
APIC Cluster Troubleshooting 727
Fabric Node Troubleshooting 734
How to Verify Physical- and Platform-Related Issues 737
Counters 737
CPU Packet Captures 743
SPAN 748
Troubleshooting Endpoint Connectivity 751
Endpoint Tracker and Log Files 752
Enhanced Endpoint Tracker (EPT) App 756
Rogue Endpoint Detection 758
Troubleshooting Contract-Related Issues 759
Verifying Policy Deny Drops 764
Embedded Logic Analyzer Module (ELAM) 765
Summary 769
Review Key Topics 769
Review Questions 769
Chapter 14 The ACI Visibility & Troubleshooting Tool 771
Visibility & Troubleshooting Tool Overview 771
Faults Tab 772
Drop/Stats Tab 773
Ingress/Egress Buffer Drop Packets 774
Ingress Error Drop Packets Periodic 774
Storm Control 774
Ingress Forward Drop Packets 775
Ingress Load Balancer Drop Packets 776
Contract Drops Tab 777
Contracts 777
Contract Considerations 778
Events and Audits Tab 779
Traceroute Tab 780
Atomic Counter Tab 782
Latency Tab 785
SPAN Tab 786
Network Insights Resources (NIR) Overview 787
Summary 790
Chapter 15 Troubleshooting Use Cases 791
Troubleshooting Fabric Discovery: Leaf Discovery 792
Troubleshooting APIC Controllers and Clusters: Clustering 795
Troubleshooting Management Access: Out-of-Band EPG 799
Troubleshooting Contracts: Traffic Not Traversing a Firewall as Expected 801
Troubleshooting Contracts: Contract Directionality 804
Troubleshooting End Host Connectivity: Layer 2 Traffic Flow Through ACI 807
Troubleshooting External Layer 2 Connectivity: Broken Layer 2 Traffic Flow Through ACI 812
Troubleshooting External Layer 3 Connectivity: Broken Layer 3 Traffic Flow Through ACI 814
Troubleshooting External Layer 3 Connectivity: Unexpected Layer 3 Traffic Flow Through ACI 816
Troubleshooting Leaf and Spine Connectivity: Leaf Issue 821
Troubleshooting VMM Domains: VMM Controller Offline 826
Troubleshooting VMM Domains: VM Connectivity Issue After Deploying the VMM Domain 829
Troubleshooting L4-L7: Deploying an L4-L7 Device 832
Troubleshooting L4-L7: Control Protocols Stop Working After Service Graph Deployment 834
Troubleshooting Multi-Pod: BUM Traffic Not Reaching Remote Pods 837
Troubleshooting Multi-Pod: Remote L3Out Not Reachable 839
Troubleshooting Multi-Site: Using Consistency Checker to Verify State at Each Site 841
Troubleshooting Programmability Issues: JSON Script Generates Error 844
Troubleshooting Multicast Issues: PIM Sparse Mode Any-Source Multicast (ASM) 846
Summary 860
Appendix A Answers to Chapter Review Questions 861
Index 873
Foreword by Ronak Desai xxix
Introduction xxx
PART I: INTRODUCTION TO ACI
Chapter 1 Fundamental Functions and Components of Cisco ACI 1
ACI Building Blocks 8
Hardware Specifications 8
ACI Key Concepts 14
Control Plane 15
Data Plane 17
VXLAN 17
Tenant 18
VRF 19
Application Profile 20
Endpoint Group 21
Contracts 22
Bridge Domain 24
External Routed or Bridged Network 25
Summary 26
Review Key Topics 26
Review Questions 27
Chapter 2 Introduction to the ACI Policy Model 31
Key Characteristics of the Policy Model 32
Management Information Tree (MIT) 33
Benefits of a Policy Model 37
Logical Constructs 37
Tenant Objects 38
VRF Objects 39
Application Profile Objects 40
Endpoint Group Objects 41
Bridge Domain and Subnet Objects 43
Bridge Domain Options 45
Contract Objects 46
Labels, Filters, and Aliases 48
Contract Inheritance 49
Contract Preferred Groups 49
vzAny 50
Outside Network Objects 51
Physical Construct 52
Access Policies 52
Switch Policies 53
Interface Policies 54
Global Policies 55
Managed Object Relationships and Policy Resolution 57
Tags 58
Default Policies 58
How a Policy Model Helps in Diagnosis 60
Summary 63
Review Key Topics 63
Review Questions 64
Chapter 3 ACI Command-Line Interfaces 67
APIC CLIs 68
NX-OS-Style CLI 68
Bash CLI 74
ACI Fabric Switch CLIs 78
iBash CLI 78
VSH CLI 81
VSH_LC CLI 83
Summary 84
Reference 84
Chapter 4 ACI Fabric Design Options 85
Physical Design 85
Single- Versus Multiple-Fabric Design 87
Multi-Pod 97
Multi-Site 116
Remote Leaf 131
Hardware and Software Support 134
ACI Multi-Pod and Remote Leaf Integration 143
Logical Design 149
Design 1: Container-as-a-Service Using the OpenShift Platform and Calico CNI 149
Design 2: Vendor-Based ERP/SAP Hana Design with ACI 165
Design 3: vBrick Digital Media Engine Design with ACI 175
Summary 180
Review Key Topics 181
Review Questions 181
Chapter 5 End Host and Network Connectivity 185
End Host Connectivity 185
VLAN Pool 186
Domain 186
Attachable Access Entity Profiles (AAEPs) 186
Switch Policies 187
Interface Policies 188
Virtual Port Channel (VPC) 191
Port Channel 197
Access Port 201
Best Practices in Configuring Access Policies 206
Compute and Storage Connectivity 207
L4/L7 Service Device Connectivity 210
Network Connectivity 213
Connecting an External Bridge Network 213
Connecting an External Routed Network 218
Diagnosing Connectivity Problems 242
Summary 245
Review Questions 245
Chapter 6 VMM Integration 249
Virtual Machine Manager (VMM) 249
VMM Domain Policy Model 250
VMM Domain Components 250
VMM Domains 250
VMM Domain VLAN Pool Association 252
VMware Integration 257
Prerequisites for VMM Integration with AVS or VDS 257
Guidelines and Limitations for VMM Integration with AVS or VDS 257
ACI VMM Integration Workflow 258
Publishing EPGs to a VMM Domain 258
Connecting Virtual Machines to the Endpoint Group Port Groups on vCenter 259
Verifying VMM Integration with the AVS or VDS 259
Microsoft SCVMM Integration 260
Mapping ACI and SCVMM Constructs 261
Mapping Multiple SCVMMs to an APIC 262
Verifying That the OpFlex Certificate Is Deployed for a Connection from the SCVMM to the APIC 262
Verifying VMM Deployment from the APIC to the SCVMM 263
OpenStack Integration 263
Extending OpFlex to the Compute Node 264
ACI with OpenStack Physical Architecture 264
OpFlex Software Architecture 265
OpenStack Logical Topology 265
Mapping OpenStack and ACI Constructs 266
Kubernetes Integration 272
Planning for Kubernetes Integration 272
Prerequisites for Integrating Kubernetes with Cisco ACI 273
Provisioning Cisco ACI to Work with Kubernetes 274
Preparing the Kubernetes Nodes 277
Installing Kubernetes and Cisco ACI Containers 279
Verifying the Kubernetes Integration 280
OpenShift Integration 281
Planning for OpenShift Integration 282
Prerequisites for Integrating OpenShift with Cisco ACI 283
Provisioning Cisco ACI to Work with OpenShift 284
Preparing the OpenShift Nodes 287
Installing OpenShift and Cisco ACI Containers 290
Updating the OpenShift Router to Use the ACI Fabric 291
Verifying the OpenShift Integration 291
VMM Integration with ACI at Multiple Locations 292
Multi-Site 292
Remote Leaf 295
Summary 298
Chapter 7 L4/L7 Service Integration 299
Service Insertion 299
The Service Graph 300
Managed Mode Versus Un-Managed Mode 301
L4-L7 Integration Use Cases 302
How Contracts Work in ACI 303
The Shadow EPG 306
Configuring the Service Graph 307
Service Graph Design and Deployment Options 312
Policy-Based Redirect (PBR) 322
PBR Design Considerations 323
PBR Design Scenarios 324
Configuring the PBR Service Graph 325
Service Node Health Check 326
Common Issues in the PBR Service Graph 328
L4/L7 Service Integration in Multi-Pod and Multi-Site 332
Multi-Pod 332
Multi-Site 338
Review Questions 342
Chapter 8 Automation and Orchestration 343
The Difference Between Automation and Orchestration 343
Benefits of Automation and Orchestration 344
REST API 349
Automating Tasks Using the Native REST API: JSON and XML 351
API Inspector 351
Object (Save As) 353
Visore (Object Store Browser) 355
MOQuery 357
Automation Use Cases 364
Automating Tasks Using Ansible 372
Ansible Support in ACI 375
Installing Ansible and Ensuring a Secure Connection 378
APIC Authentication in Ansible 382
Automation Use Cases 384
Orchestration Through UCS Director 392
Management Through Cisco UCS Director 392
Automation and Orchestration with Cisco UCS Director 393
Automation Use Cases 395
Summary 402
Review Questions 402
PART II: MONITORING AND MANAGEMENT BEST PRACTICES
Chapter 9 Monitoring ACI Fabric 405
Importance of Monitoring 405
Faults and Health Scores 407
Faults 407
Health Scores 411
ACI Internal Monitoring Tools 415
SNMP 415
Syslog 420
NetFlow 426
ACI External Monitoring Tools 430
Network Insights 430
Network Assurance Engine 437
Tetration 453
Monitoring Through the REST API 473
Monitoring an APIC 475
Monitoring Leafs and Spines 482
Monitoring Applications 499
Summary 505
Review Questions 506
Chapter 10 Network Management and Monitoring Configuration 509
Out-of-Band Management 509
Creating Static Management Addresses 510
Creating the Management Contract 510
Choosing the Node Management EPG 513
Creating an External Management Entity EPG 513
Verifying the OOB Management Configuration 515
In-Band Management 517
Creating a Management Contract 517
Creating Leaf Interface Access Policies for APIC INB Management 518
Creating Access Policies for the Border Leaf(s) Connected to L3Out 520
Creating INB Management External Routed Networks (L3Out) 522
Creating External Management EPGs 524
Creating an INB BD with a Subnet 527
Configuring the Node Management EPG 529
Creating Static Management Addresses 530
Verifying the INB Management Configuration 530
AAA 533
Configuring Cisco Secure ACS 533
Configuring Cisco ISE 542
Configuring AAA in ACI 547
Recovering with the Local Fallback User 550
Verifying the AAA Configuration 550
Syslog 551
Verifying the Syslog Configuration and Functionality 555
SNMP 556
Verifying the SNMP Configuration and Functionality 562
SPAN 566
Access SPAN 567
Fabric SPAN 571
Tenant SPAN 572
Ensuring Visibility and Troubleshooting SPAN 575
Verifying the SPAN Configuration and Functionality 576
NetFlow 577
NetFlow with Access Policies 580
NetFlow with Tenant Policies 582
Verifying the NetFlow Configuration and Functionality 585
Summary 587
PART III: ADVANCED FORWARDING AND TROUBLESHOOTING TECHNIQUES
Chapter 11 ACI Topology 589
Physical Topology 589
APIC Initial Setup 593
Fabric Access Policies 595
Switch Profiles, Switch Policies, and Interface Profiles 595
Interface Policies and Policy Groups 596
Pools, Domains, and AAEPs 597
VMM Domain Configuration 601
VMM Topology 601
Hardware and Software Specifications 603
Logical Layout of EPGs, BDs, VRF Instances, and Contracts 605
L3Out Logical Layout 606
Summary 608
Review Key Topics 608
References 609
Chapter 12 Bits and Bytes of ACI Forwarding 611
Limitations of Traditional Networks and the Evolution of Overlay Networks 611
High-Level VXLAN Overview 613
IS-IS, TEP Addressing, and the ACI Underlay 615
IS-IS and TEP Addressing 615
FTags and the MDT 618
Endpoint Learning in ACI 626
Endpoint Learning in a Layer 2-Only Bridge Domain 627
Endpoint Learning in a Layer 3-Enabled Bridge Domain 635
Fabric Glean 640
Remote Endpoint Learning 641
Endpoint Mobility 645
Anycast Gateway 647
Virtual Port Channels in ACI 649
Routing in ACI 651
Static or Dynamic Routes 651
Learning External Routes in the ACI Fabric 656
Transit Routing 659
Policy Enforcement 661
Shared Services 664
L3Out Flags 668
Quality of Service (QoS) in ACI 669
Externally Set DSCP and CoS Markings 671
CoS Preservation in ACI 672
Multi-Pod 674
Multi-Site 680
Remote Leaf 684
Forwarding Scenarios 686
ARP Flooding 686
Layer 2 Known Unicast 688
ARP Optimization 690
Layer 2 Unknown Unicast Proxy 690
L3 Policy Enforcement When Going to L3Out 693
L3 Policy Enforcement for External Traffic Coming into the Fabric 695
Route Leaking/Shared Services 695
Consumer to Provider 695
Provider to Consumer 698
Multi-Pod Forwarding Examples 698
ARP Flooding 700
Layer 3 Proxy Flow 700
Multi-Site Forwarding Examples 703
ARP Flooding 703
Layer 3 Proxy Flow 705
Remote Leaf 707
ARP Flooding 707
Layer 3 Proxy Flow 710
Summary 713
Review Key Topics 713
References 714
Review Questions 714
Chapter 13 Troubleshooting Techniques 717
General Troubleshooting 717
Faults, Events, and Audits 718
moquery 722
iCurl 724
Visore 726
Infrastructure Troubleshooting 727
APIC Cluster Troubleshooting 727
Fabric Node Troubleshooting 734
How to Verify Physical- and Platform-Related Issues 737
Counters 737
CPU Packet Captures 743
SPAN 748
Troubleshooting Endpoint Connectivity 751
Endpoint Tracker and Log Files 752
Enhanced Endpoint Tracker (EPT) App 756
Rogue Endpoint Detection 758
Troubleshooting Contract-Related Issues 759
Verifying Policy Deny Drops 764
Embedded Logic Analyzer Module (ELAM) 765
Summary 769
Review Key Topics 769
Review Questions 769
Chapter 14 The ACI Visibility & Troubleshooting Tool 771
Visibility & Troubleshooting Tool Overview 771
Faults Tab 772
Drop/Stats Tab 773
Ingress/Egress Buffer Drop Packets 774
Ingress Error Drop Packets Periodic 774
Storm Control 774
Ingress Forward Drop Packets 775
Ingress Load Balancer Drop Packets 776
Contract Drops Tab 777
Contracts 777
Contract Considerations 778
Events and Audits Tab 779
Traceroute Tab 780
Atomic Counter Tab 782
Latency Tab 785
SPAN Tab 786
Network Insights Resources (NIR) Overview 787
Summary 790
Chapter 15 Troubleshooting Use Cases 791
Troubleshooting Fabric Discovery: Leaf Discovery 792
Troubleshooting APIC Controllers and Clusters: Clustering 795
Troubleshooting Management Access: Out-of-Band EPG 799
Troubleshooting Contracts: Traffic Not Traversing a Firewall as Expected 801
Troubleshooting Contracts: Contract Directionality 804
Troubleshooting End Host Connectivity: Layer 2 Traffic Flow Through ACI 807
Troubleshooting External Layer 2 Connectivity: Broken Layer 2 Traffic Flow Through ACI 812
Troubleshooting External Layer 3 Connectivity: Broken Layer 3 Traffic Flow Through ACI 814
Troubleshooting External Layer 3 Connectivity: Unexpected Layer 3 Traffic Flow Through ACI 816
Troubleshooting Leaf and Spine Connectivity: Leaf Issue 821
Troubleshooting VMM Domains: VMM Controller Offline 826
Troubleshooting VMM Domains: VM Connectivity Issue After Deploying the VMM Domain 829
Troubleshooting L4-L7: Deploying an L4-L7 Device 832
Troubleshooting L4-L7: Control Protocols Stop Working After Service Graph Deployment 834
Troubleshooting Multi-Pod: BUM Traffic Not Reaching Remote Pods 837
Troubleshooting Multi-Pod: Remote L3Out Not Reachable 839
Troubleshooting Multi-Site: Using Consistency Checker to Verify State at Each Site 841
Troubleshooting Programmability Issues: JSON Script Generates Error 844
Troubleshooting Multicast Issues: PIM Sparse Mode Any-Source Multicast (ASM) 846
Summary 860
Appendix A Answers to Chapter Review Questions 861
Index 873