1
Systems in Data Centers

There are different types of IT equipment that serve different functions depending on customer application. This chapter provides an overview of servers, storage arrays, switches and their components.

1.1. Servers

A server is a broad term describing a specific piece of IT equipment that provides computing capability and runs software applications in an environment networked with other IT equipment, including other servers. Most servers contain the following major hardware building blocks: processors, memory, chipset, input/output (I/O) devices, storage, peripherals, voltage regulators (VRs), power supplies and cooling systems. Additional application-specific integrated circuits (ASICs) may be necessary, such as an onboard redundant array of independent disks (RAID) controller and a server management controller.

Volume rack-mounted servers are designed to fit within commonly available rack sizes, such as the 19 in. (0.5 m) rack form factor defined by EIA/ECA Standard 310-E specification1. The vertical dimension is expressed in terms of rack units or just units (U). One U or 1U represents 1.75 in. (44.45 mm) of vertical height within a rack. Servers used for computing are available in standard rack-mount and custom configurations. Typical dimensions and sizes for standard rack-mount compute servers are full-width 1U, 2U or 4U. A single-server chassis may contain multiple server nodes. Each node is defined as containing all key components, except power supplies, needed to make up a complete server. These nodes simply share the larger chassis infrastructure to conserve data center space. For more dense servers, there are 1U and 2U server enclosures that house several 1U ½-width servers.

Microservers are an emerging technology. They are based on system on a chip (SOC) design where all the functions, which are located on a motherboard for a classic server, are integrated on a single chip with the exception of memory, boot flash and power circuits. SOC are usually less power hungry than usual microprocessors leading to microservers that are more dense than classic servers. Although microservers and SOC are not analyzed in the following chapters, they are worth mentioning. These servers generally provide sufficient, targeted performance with optimized performance-per-watt capability, while being easily scalable with shared power and cooling infrastructure for individual servers.

To achieve even higher compute density than the 1U form factor, blade servers are another option. Each manufacturer designs their blades based on their own packaging and design goals. These blade chassis range from 3U to 10U tall and can house many blades. Blade servers are the result of technology compaction, which allows for a greater processing density in the same equipment volume. The greater processing density also results in greater power and heat density, further complicating data center cooling. Server components that had previously been packaged inside the tower/pedestal or rack-mounted server (e.g. fans and power supplies) are still required, but these components are now located within a chassis (or enclosure) that is designed to house multiple blade servers in a side-by-side or stacked configuration. Most of the time, the blade chassis includes networking, management and even storage functions for the blade servers, while the blade server integrates at least one controller (Ethernet, fiber channel, etc.) on the motherboard. Extra interfaces can be added using mezzanine cards.

Figure 1.1 illustrates 1U, 2U, 4U full width servers, a 2U chassis hosting four ½-width nodes and a 5U blade chassis hosting eight blades.

Examples of such servers will be given in Chapter 2.

Figure 1.1. 1U, 2U, 4U full width servers, 2U chassis with four ½-width nodes and a 5U blade chassis with eight blades including a 1U base for power supplies

1.2. Storage arrays

Disk arrays are enclosures that provide ample non-volatile storage space for use by servers. Like servers, and depending on the scale of deployment required, the storage configuration may be a standard rack-mounted system with varying unit height or possibly a custom stand-alone piece of equipment. Disk storage arrays are typically designed for use in EIA/ECA Standard-310-E-compliant racks. The enclosure contains the storage in either small or large form factor drives in addition to the controllers, midplane and batteries for cache backup. The storage array enclosure typically uses redundant power supplies and cooling in the event of component failures. One of the more challenging aspects of storage arrays is the design of a battery backup system to prevent data loss in case of a power interruption or loss. For an in-depth discussion of storage array thermal guidelines, please consult the ASHRAE storage equipment white paper (ASHRAE TC 9.9 2015).

While disk storage arrays are typically used for online storage, backup and disaster recovery, tape storage is known for its low cost and longevity for archiving purposes. Tape systems come in a variety of different formats based on the type of tape media.

1.3. Data center networking

A computer network, or simply a network, is a collection of computers and other hardware interconnected by communication channels that allow sharing of resources and information. Networking equipment facilitates the interconnection of devices and the sharing of data both within the data center and beyond. Networks tend to be designed in a hierarchical manner, with many devices (such as servers and storage devices in the case of a data center) connected to a switch that is connected to another switch at the next level of the hierarchy, and so on. Another common topology is a mesh configuration in which many peer network switches are connected to one another to form a single level of the hierarchy. We will consider three different levels of a network hierarchy: core, distribution and top of rack (TOR). More elaborate configurations will not be covered. The core network function can be thought of as the gateway through which all data entering and exiting the data center must pass. As such, the core network equipment is connected either directly or indirectly to every device in the data center. The core switch is also connected to a service provider, which is the "pipe" through which all data passes from the data center to the Internet. The distribution level of the network acts as an intermediate level between the core and edge levels of the network, and as such can offload some of the work the core network equipment needs to do.

Specifically, the distribution level is useful for passing data between machines inside the data center or aggregating ports to reduce the number of physical ports required at the core. The TOR network level consists of switches that connect directly to devices that are generating or consuming the data, and then pass the data up to the distribution or core level. A data center network implementation may have all three of these levels, or it may combine or eliminate some of the levels. In large data centers, the load on the networking equipment can be substantial, both in terms of port count and data throughput. The end-of-row (EOR) distribution equipment offloads the core equipment in both throughput and port count. The TOR edge networking equipment offloads the distribution equipment in both throughput and port count, in the same way as the distribution networking equipment offloads the core networking equipment. Switches enable communication between devices connected on a network. In the case of a data center, servers and storage arrays are connected to multiple switches in a hierarchical manner.

The ASIC chip decides where that data need to go and sends it back out through the correct port. The central processing unit (CPU) controls both the PHY and the ASIC. The CPU can take data from the network, process it and then send it back out onto the network.

1.4. Components

1.4.1. Central processing unit

The processor, also referred to as the CPU, is one of the greatest sources of heat generation within a server. Aside from the basic processing of data and instructions to provide an output result, processors may also have many more features for managing data and power throughout a system. The processor die is generally housed in a package that includes a substrate (i.e. a small printed circuit board, or PCB, for bringing out signals) and a lid, as shown in Figure 2.1. The lid, or case, more evenly distributes heat to an attached cooling component such as a heat sink (air-cooled) or cold plate (liquid-cooled), as shown in Figure 2.1. In most cases, a socket is used to enable removal and replacement of the processor on the motherboard. Some lower powered processors are lidless using direct attachment of a heat sink on top of the die. In the volume server segment, limitations of low-cost thermal solutions and a greater focus on performance per watt have slowed the generational increases in thermal design power (TDP). The transition to multicore processors has maintained Moore's law (a doubling of the transistors on a chip every 18 months), an improvement even within this limited power envelope. However, recent trends, which include the integration of functions previously implemented in a chipset or external devices, greater numbers of high-performance I/O interfaces and memory channels, larger internal caches and incorporation of...