Table of Figures

1.1 High-tech revolution 1.2 Neural network development timeline 1.3 ImageNet challenge 1.4 Neural network model 1.5 Regression 1.6 Clustering 1.7 Neural network top 1 accuracy vs. computational complexity 1.8 Neural network top 1 accuracy density vs. model efficiency [14] 1.9 Neural network memory utilization and computational complexity [14] 2.1 Deep neural network AlexNet architecture [1] 2.2 Deep neural network AlexNet model parameters 2.3 Deep neural network AlexNet feature map evolution [3] 2.4 Convolution function 2.5 Nonlinear activation functions 2.6 Pooling functions 2.7 Dropout layer 2.8 Deep learning hardware issues [1] 3.1 Intel Xeon processor ES 2600 family Grantley platform ring architecture [3] 3.2 Intel Xeon processor scalable family Purley platform mesh architecture [3] 3.3 Two-socket configuration 3.4 Four-socket ring configuration 3.5 Four-socket crossbar configuration 3.6 Eight-socket configuration 3.7 Sub-NUMA cluster domains [3] 3.8 Cache hierarchy comparison 3.9 Intel multiple sockets parallel processing 32 3.10 Intel multiple socket training performance comparison [4] 3.11 Intel AVX-512 16 bits FMA operations (VPMADDWD?+?VPADDD) 3.12 Intel AVX-512 with VNNI 16 bits FMA operation (VPDPWSSD) 3.13 Intel low-precision convolution 3.14 Intel Xenon processor training throughput comparison [2] 3.15 Intel Xenon processor inference throughput comparison [2] 3.16 NVIDIA turing GPU architecture 3.17 NVIDIA GPU shared memory 3.18 Tensor core 4 × 4 × 4 matrix operation [9] 3.19 Turing tensor core performance [7] 3.20 Matrix D thread group indices 3.21 Matrix D 4 × 8 elements computation 3.22 Different size matrix multiplication 3.23 Simultaneous multithreading (SMT) 3.24 Multithreading schedule 3.25 GPU with HBM2 architecture 3.26 Eight GPUs NVLink2 configuration 3.27 Four GPUs NVLink2 configuration 3.28 Two GPUs NVLink2 configuration 3.29 Single GPU NVLink2 configuration 3.30 NVDLA core architecture 3.31 NVDLA small system model 3.32 NVDLA large system model 3.33 NVDLA software dataflow 3.34 Tensor processing unit architecture 3.35 Tensor processing unit floorplan 3.36 Multiply-Accumulate (MAC) systolic array 3.37 Systolic array matrix multiplication 3.38 Cost of different numerical format operation 3.39 TPU brain floating-point format 3.40 CPU, GPU, and TPU performance comparison [15] 3.41 Tensor Processing Unit (TPU) v1 3.42 Tensor Processing Unit (TPU) v2 3.43 Tensor Processing Unit (TPU) v3 3.44 Google TensorFlow subgraph optimization 3.45 Microsoft Brainwave configurable cloud architecture 3.46 Tour network topology 3.47 Microsoft Brainwave design flow 3.48 The Catapult fabric shell architecture 3.49 The Catapult fabric microarchitecture 3.50 Microsoft low-precision quantization [27] 66 3.51 Matrix-vector multiplier overview 3.52 Tile engine architecture 3.53 Hierarchical decode and dispatch scheme 3.54 Sparse matrix-vector multiplier architecture 3.55 (a) Sparse Matrix; (b) CSR Format; and (c) CISR Format 4.1 Data streaming TCS model 4.2 Blaize depth-first scheduling approach 4.3 Blaize graph streaming processor architecture 4.4 Blaize GSP thread scheduling 4.5 Blaize GSP instruction scheduling 4.6 Streaming vs. sequential processing comparison 4.7 Blaize GSP convolution operation 4.8 Intelligence processing unit architecture [8] 4.9 Intelligence processing unit mixed-precision multiplication 4.10 Intelligence processing unit single-precision multiplication 4.11 Intelligence processing unit interconnect architecture [9] 4.12 Intelligence processing unit bulk synchronous parallel model 4.13 Intelligence processing unit bulk synchronous parallel execution trace [9] 4.14 Intelligence processing unit bulk synchronous parallel inter-chip execution [9] 5.1 Deep convolutional neural network hardware architecture 5.2 Convolution computation 5.3 Filter decomposition with zero padding 5.4 Filter decomposition approach 5.5 Data streaming architecture with the data flow 5.6 DCNN accelerator COL buffer architecture 5.7 Data streaming architecture with 1×1 convolution mode 5.8 Max pooling architecture 5.9 Convolution engine architecture 5.10 Accumulation (ACCU) buffer architecture 5.11 Neural network model compression 5.12 Eyeriss system architecture 5.13 2D convolution to 1D multiplication mapping 5.14 2D convolution to 1D multiplication - step #1 5.15 2D convolution to 1D multiplication - step #2 5.16 2D convolution to 1D multiplication - step #3 5.17 2D convolution to 1D multiplication - step...