1
Fundamentals of Distributed Storage

As the digital economy emerges as a new driver for economic and social development, accelerating the realization of data as a production factor has become an essential requirement to promote the digital economy and facilitate high-quality development. The advent of new technologies such as 5G, cloud computing, artificial intelligence (AI), high-performance computing (HPC), and big data has given rise to numerous emerging applications, leading to an explosive growth of data volume. The aggregation, circulation, and application of massive data have placed higher demands on data storage systems. Distributed storage, as a critical infrastructure, represents an optimal solution for substantial volumes of data. Therefore, it is imperative to expedite research and development efforts related to super-large-scale, highly reliable, highly available, and high-performance distributed storage in order to adapt to the expanding industrial spaces and evolving application scenarios.

This chapter first introduces the evolution and characteristics of distributed storage and then focuses on the concept of data reliability, as well as on the common fault-tolerant mechanisms. Finally, it discusses the future trends of distributed storage from a macro perspective.

1.1 Introduction to Distributed Storage

More than 60 years ago, the first-generation computer ENIAC (McCartney 1999, pp. 1-11) could execute over 5,000 additions or more than 400 multiplications per second with only 100 B of magnetic core storage capacity. However, a compressed digital photo with a resolution of 3,600 × 2,700 pixels is approximately 4 MB in size, while a photograph of the spectacular spiral galaxy M81 captured by the Hubble Space Telescope occupies 689 MB. The storage capacity required for high-definition movie playback is around 20 GB, and this necessitates an input/output (I/O) sustained data transfer rate of at least 3 MB/s. In addition to capacity requirements, many applications impose stringent demands on response time, throughput, transmission bandwidth, and so on. These escalating demands propel the evolution of storage systems from small to large, slow to fast, simple to complex, and single-machine to a distributed system; they also present higher-level requirements for robustness, availability, reliability, and performance.

1.1.1 Evolution of Storage Systems

Generalized storage devices include CPU registers, multi-level caches, memory, and the external storage system. The first three are also known as the memory system. In a narrow sense, storage systems usually refer to external storage systems. This book also mainly centers on the external storage systems.

The data units of a computer are byte (B), kilobyte (KB), megabyte (MB), gigabyte (GB), terabyte (TB), petabyte (PB), exabyte (EB), zettabyte (ZB), and so on. One byte represents one character of information and is composed of eight binary bits. The conversion relations between the units are as follows: 1 KB is equal to 1024 B, 1 MB is equal to 1024 KB, 1 GB is equal to 1024 MB, and 1 TB is equal to 1024 GB.

If users usually only create simple documents or spreadsheets, then most files may be a few hundred KBs or occasionally a few MBs in size. However, if the computer is used to store and process digital photos, then the average file size will be in the range of several MBs. When the computer is used to edit and store videos, the size of a single file may be tens of MB or even several GB. For example, one hour of DV format video footage consumes approximately 12 GB; one minute of uncompressed 1,920 × 1,080 high-definition video is approximately 9 GB. Therefore, adequate and scalable storage capacity is very important.

Apart from capacity, the update and access frequency of stored data is also a key factor to be considered in the selection of external storage devices. For example, digital photographers may have incremental storage capacity requirements. With each shoot, they will add several hundred MB or several GB of photos as backups to the storage system and require these photos to be stored permanently without being altered. In contrast, three-dimensional (3D) computer animation may require regularly repeated access to several GB of data and replace the previously stored data. In this case, not only rewritable storage media is needed, but higher requirements are also imposed on the response speed. It is these various requirements that have given rise to different storage systems. Next, the development of storage systems is introduced in chronological order.

1. Hard Disk:

   Most desktop computers and laptops nowadays still rely on rotational hard disks as high-capacity storage devices for storing the operating system kernel, applications, user data, and so on. Traditional rotational hard disk drives consist of one or more disks that are stacked together and covered with surface magnetic media that can be driven by write heads and read heads. Hard disk drives can transfer data directly to other computer hardware through three interface types: SATA, IDE/UDMA, or SCSI, and their speed ranges from 4,200 to 15,000 rpm.

2. Redundant Array of Independent Disks:

   Redundant array of independent disks (RAID) (Perumal and Kritzinger 2004, pp. 1-9) refers to the use of multiple hard disks connected together in servers and high-end personal computer workstations and to the operation of storing data through multiple physical drives. There are many available RAID configurations, of which the three basic ones are explained as follows:

   1. RAID 0 divides a data file into blocks and stores them on two or more hard disks, with each hard disk storing the data blocks evenly. Parallel read operations can be used to improve system performance without increasing capacity.
   2. RAID 1 employs data mirroring to symmetrically distribute data blocks across two or more disks, effectively reducing the risk of disk faults. This redundancy ensures that in the event of a disk fault, the complete dataset can still be retrieved from other disks.
   3. RAID 5 requires at least three disks. Similar to RAID 0, it not only spans multiple disks to store different data blocks but also simultaneously stores parity blocks for these data blocks. This configuration ensures that in the event of a single disk fault, the failed data can be recovered using the parity blocks stored on other disk drives.

   Other RAID configurations are generally further optimized or combined from the basic configurations mentioned earlier, such as increasing the fault-tolerant level of the disks and expanding to disk arrays across different regions.

3. Distributed Storage:

   The exponential growth in data volume has necessitated more rigorous storage requirements, including scalable storage capacity, enhanced reliability, increased throughput, and reduced cost. Traditional single-machine storage such as RAID is no longer suitable for large-scale data. Distributed storage systems have emerged as a viable solution. Generic distributed storage involves the dispersion of data across multiple servers in a cluster of storage nodes, and the synchronization or access of data through a distributed network between the nodes. Data is spread across multiple nodes, and reliability is maintained through specific redundant mechanisms. Specifically speaking, a distributed storage system typically comprises the following components:

   1. Client: Client encompasses the user interface and is responsible for engaging with users, receiving their requests, and delivering responses.
   2. Network Link: The network link plays a crucial role in transmitting diverse data, including user requests, read and write operations, and responses.
   3. Storage Node: Storage node, as a fundamental component of the distributed storage system, plays a pivotal role in both actual data storage and the processing of read and write operations.
   4. Metadata Management Node: Metadata management node is tasked with overseeing metadata information of the distributed storage system, encompassing data file storage locations, copy counts, and access permissions. It also facilitates load balancing and data migration of storage nodes to optimize system performance.
4. Cloud Storage:

   Cloud storage (Ghani et al. 2020, pp. 1-21) represents highly scalable distributed storage that disperses data across diverse geographical locations. It typically adopts a distributed storage architecture to cater to users' requirements for data storage. Unlike traditional distributed storage, which is limited by geography and network bandwidth, cloud storage is typically offered by global cloud service providers, allowing users to directly access data via the Internet. Therefore, cloud storage can be viewed as an extension of distributed storage on the Internet, delivering a more efficient, dependable, and adaptable solution.

1.1.2 Characteristics of Distributed Storage

A distributed storage system can provide storage services by interconnecting multiple storage nodes, which are composed of soft and hardware components. Compared with traditional centralized storage, distributed storage distributes data across multiple relatively independent storage nodes to achieve higher response speed and scales the capacity by increasing the number of storage nodes. It also...