I/O Consolidation in the Data Center
Published on 17. September 2009
Book
Paperback/Softback
168 pages
978-1-58705-888-2 (ISBN)
Description
Using Fibre Channel over Ethernet (FCoE) and related technologies, data centers can consolidate data traffic onto a single network switch, simplifying their environments, promoting virtualization, and substantially reducing power and cooling costs. This emerging technology is drawing immense excitement, but few enterprise IT decision-makers and implementers truly understand it. I/O Consolidation in the Data Center is the only complete, up-to-date guide to FCoE. FCoE innovators Silvano Gai and Claudio DeSanti (chair of the T11 FCoE standards working group) systematically explain the technology: its benefits, tradeoffs, and what it will take to implement it successfully in production environments. Unlike most other discussions of FCoE, this book fully reflects the final, recently-approved industry standard. The authors also present five detailed case studies illustrating typical FCoE adoption scenarios, as well as an extensive Q and A section addressing the issues enterprise IT professionals raise most often. This is a fully updated version of Silvano Gai's privately-published book on FCoE, written for leading FCoE pioneer Nuova Systems before the company was acquired by Cisco. Nearly 12,000 copies of that book have already been distributed, demonstrating the immense interest in FCoE technology, and the scarcity of reliable information that has existed about it.
More details
Persons
Silvano Gai,
who grew up in a small village near Asti, Italy, has more than 27 years of experience
in computer engineering and computer networks. He is the author of several books and technical
publications on computer networking and multiple Internet Drafts and RFCs. He is responsible
for 30 issued patents and 50 patent applications. His background includes 7 years as a full
professor of computer engineering, tenure track, at Politecnico di Torino, Italy, and seven years
as a researcher at the CNR (Italian National Council for Scientifi c Research). For the past 12
years, he has been in Silicon Valley where, in the position of Cisco Fellow, he was an architect
of the Cisco Catalyst family of network switches, of the Cisco MDS family of storage networking
switches, and of the Nexus family of data center switches. Silvano teaches a course on I/O
Consolidation, Data Center Ethernet, and Fibre Channel over Ethernet at Stanford University
(see http://scpd.stanford.edu/certifi cates/fcoe).
Claudio DeSanti is a Distinguished Engineer in the Advanced Architectures & Research organization
at Cisco. He represents Cisco in several National and International Standards Bodies,
such as INCITS Technical Committee T11, IEEE 802.1, IETF, and in industry associations. He
is vice chairman of the INCITS T11 Technical Committee, chairperson of various working
groups, including FC-BB-5, where FCoE has been developed, and technical editor of different
standards, including IEEE 802.1Qbb, where Priority-based Flow Control is defi ned. He is
author of several patents and international publications, including ten RFCs in IETF and other
standards in the American National Standard Institute. He received many honors and awards,
including the INCITS 2008 Technical Excellence Award, the INCITS 2007 Team Award, and the
INCITS 2006 Gene Milligan Award for Effective Committee Management. Claudio's research
interests include network protocols, storage networking, routing, and security. He holds a Ph.D.
in computer engineering from the Scuola Superiore Sant'Anna in Pisa, Italy.
who grew up in a small village near Asti, Italy, has more than 27 years of experience
in computer engineering and computer networks. He is the author of several books and technical
publications on computer networking and multiple Internet Drafts and RFCs. He is responsible
for 30 issued patents and 50 patent applications. His background includes 7 years as a full
professor of computer engineering, tenure track, at Politecnico di Torino, Italy, and seven years
as a researcher at the CNR (Italian National Council for Scientifi c Research). For the past 12
years, he has been in Silicon Valley where, in the position of Cisco Fellow, he was an architect
of the Cisco Catalyst family of network switches, of the Cisco MDS family of storage networking
switches, and of the Nexus family of data center switches. Silvano teaches a course on I/O
Consolidation, Data Center Ethernet, and Fibre Channel over Ethernet at Stanford University
(see http://scpd.stanford.edu/certifi cates/fcoe).
Claudio DeSanti is a Distinguished Engineer in the Advanced Architectures & Research organization
at Cisco. He represents Cisco in several National and International Standards Bodies,
such as INCITS Technical Committee T11, IEEE 802.1, IETF, and in industry associations. He
is vice chairman of the INCITS T11 Technical Committee, chairperson of various working
groups, including FC-BB-5, where FCoE has been developed, and technical editor of different
standards, including IEEE 802.1Qbb, where Priority-based Flow Control is defi ned. He is
author of several patents and international publications, including ten RFCs in IETF and other
standards in the American National Standard Institute. He received many honors and awards,
including the INCITS 2008 Technical Excellence Award, the INCITS 2007 Team Award, and the
INCITS 2006 Gene Milligan Award for Effective Committee Management. Claudio's research
interests include network protocols, storage networking, routing, and security. He holds a Ph.D.
in computer engineering from the Scuola Superiore Sant'Anna in Pisa, Italy.
Content
I/O Consolidation 1
Introduction 1
What Is I/O Consolidation 2
Merging the Requirements 3
Why I/O Consolidation Has Not Yet Been Successful 4
Fundamental Technologies 5
PCI-Express 5
10 Gigabit Ethernet 5
Additional Requirements 8
Buffering Requirements 8
Layer 2 Only 9
Switch Architecture 9
Low Latency 10
Native Support for Storage Traffi c 11
RDMA Support 11
Enabling Technologies 15
Introduction 15
Lossless Ethernet 15
PAUSE 15
Credits Versus PAUSE 17
PAUSE Propagation 18
Is Lossless Better? 19
Why PAUSE Is Not Widely Deployed 20
Priority-based Flow Control (PFC) 20
Additional Components 22
DCBX: Data Center Bridging eXchange 22
Bandwidth Management 23
Congestion Management 25
Delayed Drop 26
Going Beyond Spanning Tree 28
Active-Active Connectivity 32
Etherchannel 32
Virtual Switching System (VSS) 32
virtual Port Channel (vPC) 34
Ethernet Host Virtualizer 36
Layer 2 Multipath (L2MP) 38
Basic Mechanisms in L2MP 40
Cisco DBridges 47
IETF RBridges and the TRILL Project 51
VEB: Virtual Ethernet Bridging 52
Server Virtualization 53
SR-IOV 54
The IEEE Standard Effort 54
VEB in the Adapter 55
VEB in the Switch 56
VNTag 57
Fabric Extenders 59
VN-Link 60
Questions and Answers 63
Does FCoE Uses Credits? 63
High Availability of PAUSE and Credits 63
Queue Size 63
Long-Haul 63
FECN/BECN 64
Confi guration 64
Bandwidth Prioritization 64
Storage Bandwidth 64
Cisco DCB/FCoE Support 65
10GE NICs 65
IP Routing 65
Lossless Ethernet Versus Infi niband 66
Nomenclature 66
Fibre Channel over Ethernet 67
Introduction 67
Fibre Channel 69
Fibre Channel Architectural Models 71
FCoE Mapping 73
FCoE Architectural Models 74
FCoE Benefi ts 79
FCoE Data Plane 80
FCoE Topologies 82
FCoE Addressing 85
FCoE Forwarding 87
FPMAs and SPMAs 90
FIP: FCoE Initialization Protocol 92
FIP Messages 93
FIP VLAN Discovery 97
FIP Discovery 98
FIP Virtual Link Instantiation 103
FIP Virtual Link Maintenance 108
Converged Network Adapters 110
FCoE Open Software 112
Network Tools 113
FCoE and Virtualization 114
Fibre Channel Block I/O 115
iSCSI Block I/O 116
Moving a VM 117
FCoE and Block I/O 117
FCoE FAQ 118
Is FCoE Routable? 118
iSCSI Versus FCoE? 120
Does FCoE Require Gateways? 123
Case Studies 125
Introduction 125
I/O Consolidation with Discrete Servers 126
Top-of-Rack Consolidated I/O 129
Example with Blade Servers 131
Updating the Distribution Layer 133
Unifi ed Computing System 136
Bibliography 139
PCI Express 139
IEEE 802.3 139
IEEE 802.1 139
Ethernet Improvements 139
Fibre Channel 139
FCoE 140
TRILL 140
Virtualization 140
Glossary 141
Figures 145
Tables 149
Index 151
Introduction 1
What Is I/O Consolidation 2
Merging the Requirements 3
Why I/O Consolidation Has Not Yet Been Successful 4
Fundamental Technologies 5
PCI-Express 5
10 Gigabit Ethernet 5
Additional Requirements 8
Buffering Requirements 8
Layer 2 Only 9
Switch Architecture 9
Low Latency 10
Native Support for Storage Traffi c 11
RDMA Support 11
Enabling Technologies 15
Introduction 15
Lossless Ethernet 15
PAUSE 15
Credits Versus PAUSE 17
PAUSE Propagation 18
Is Lossless Better? 19
Why PAUSE Is Not Widely Deployed 20
Priority-based Flow Control (PFC) 20
Additional Components 22
DCBX: Data Center Bridging eXchange 22
Bandwidth Management 23
Congestion Management 25
Delayed Drop 26
Going Beyond Spanning Tree 28
Active-Active Connectivity 32
Etherchannel 32
Virtual Switching System (VSS) 32
virtual Port Channel (vPC) 34
Ethernet Host Virtualizer 36
Layer 2 Multipath (L2MP) 38
Basic Mechanisms in L2MP 40
Cisco DBridges 47
IETF RBridges and the TRILL Project 51
VEB: Virtual Ethernet Bridging 52
Server Virtualization 53
SR-IOV 54
The IEEE Standard Effort 54
VEB in the Adapter 55
VEB in the Switch 56
VNTag 57
Fabric Extenders 59
VN-Link 60
Questions and Answers 63
Does FCoE Uses Credits? 63
High Availability of PAUSE and Credits 63
Queue Size 63
Long-Haul 63
FECN/BECN 64
Confi guration 64
Bandwidth Prioritization 64
Storage Bandwidth 64
Cisco DCB/FCoE Support 65
10GE NICs 65
IP Routing 65
Lossless Ethernet Versus Infi niband 66
Nomenclature 66
Fibre Channel over Ethernet 67
Introduction 67
Fibre Channel 69
Fibre Channel Architectural Models 71
FCoE Mapping 73
FCoE Architectural Models 74
FCoE Benefi ts 79
FCoE Data Plane 80
FCoE Topologies 82
FCoE Addressing 85
FCoE Forwarding 87
FPMAs and SPMAs 90
FIP: FCoE Initialization Protocol 92
FIP Messages 93
FIP VLAN Discovery 97
FIP Discovery 98
FIP Virtual Link Instantiation 103
FIP Virtual Link Maintenance 108
Converged Network Adapters 110
FCoE Open Software 112
Network Tools 113
FCoE and Virtualization 114
Fibre Channel Block I/O 115
iSCSI Block I/O 116
Moving a VM 117
FCoE and Block I/O 117
FCoE FAQ 118
Is FCoE Routable? 118
iSCSI Versus FCoE? 120
Does FCoE Require Gateways? 123
Case Studies 125
Introduction 125
I/O Consolidation with Discrete Servers 126
Top-of-Rack Consolidated I/O 129
Example with Blade Servers 131
Updating the Distribution Layer 133
Unifi ed Computing System 136
Bibliography 139
PCI Express 139
IEEE 802.3 139
IEEE 802.1 139
Ethernet Improvements 139
Fibre Channel 139
FCoE 140
TRILL 140
Virtualization 140
Glossary 141
Figures 145
Tables 149
Index 151