Learning in Energy-Efficient Neuromorphic Computing: Algorithm and Architecture Co-Design
1st Edition
Published on 22. November 2019
Book
Hardback
280 pages
978-1-119-50738-3 (ISBN)
Description
Explains current co-design and co-optimization methodologies for building hardware neural networks and algorithms for machine learning applications
This book focuses on how to build energy-efficient hardware for neural networks with learning capabilities--and provides co-design and co-optimization methodologies for building hardware neural networks that can learn. Presenting a complete picture from high-level algorithm to low-level implementation details, Learning in Energy-Efficient Neuromorphic Computing: Algorithm and Architecture Co-Design also covers many fundamentals and essentials in neural networks (e.g., deep learning), as well as hardware implementation of neural networks.
The book begins with an overview of neural networks. It then discusses algorithms for utilizing and training rate-based artificial neural networks. Next comes an introduction to various options for executing neural networks, ranging from general-purpose processors to specialized hardware, from digital accelerator to analog accelerator. A design example on building energy-efficient accelerator for adaptive dynamic programming with neural networks is also presented. An examination of fundamental concepts and popular learning algorithms for spiking neural networks follows that, along with a look at the hardware for spiking neural networks. Then comes a chapter offering readers three design examples (two of which are based on conventional CMOS, and one on emerging nanotechnology) to implement the learning algorithm found in the previous chapter. The book concludes with an outlook on the future of neural network hardware.
* Includes cross-layer survey of hardware accelerators for neuromorphic algorithms
* Covers the co-design of architecture and algorithms with emerging devices for much-improved computing efficiency
* Focuses on the co-design of algorithms and hardware, which is especially critical for using emerging devices, such as traditional memristors or diffusive memristors, for neuromorphic computing
Learning in Energy-Efficient Neuromorphic Computing: Algorithm and Architecture Co-Design is an ideal resource for researchers, scientists, software engineers, and hardware engineers dealing with the ever-increasing requirement on power consumption and response time. It is also excellent for teaching and training undergraduate and graduate students about the latest generation neural networks with powerful learning capabilities.
This book focuses on how to build energy-efficient hardware for neural networks with learning capabilities--and provides co-design and co-optimization methodologies for building hardware neural networks that can learn. Presenting a complete picture from high-level algorithm to low-level implementation details, Learning in Energy-Efficient Neuromorphic Computing: Algorithm and Architecture Co-Design also covers many fundamentals and essentials in neural networks (e.g., deep learning), as well as hardware implementation of neural networks.
The book begins with an overview of neural networks. It then discusses algorithms for utilizing and training rate-based artificial neural networks. Next comes an introduction to various options for executing neural networks, ranging from general-purpose processors to specialized hardware, from digital accelerator to analog accelerator. A design example on building energy-efficient accelerator for adaptive dynamic programming with neural networks is also presented. An examination of fundamental concepts and popular learning algorithms for spiking neural networks follows that, along with a look at the hardware for spiking neural networks. Then comes a chapter offering readers three design examples (two of which are based on conventional CMOS, and one on emerging nanotechnology) to implement the learning algorithm found in the previous chapter. The book concludes with an outlook on the future of neural network hardware.
* Includes cross-layer survey of hardware accelerators for neuromorphic algorithms
* Covers the co-design of architecture and algorithms with emerging devices for much-improved computing efficiency
* Focuses on the co-design of algorithms and hardware, which is especially critical for using emerging devices, such as traditional memristors or diffusive memristors, for neuromorphic computing
Learning in Energy-Efficient Neuromorphic Computing: Algorithm and Architecture Co-Design is an ideal resource for researchers, scientists, software engineers, and hardware engineers dealing with the ever-increasing requirement on power consumption and response time. It is also excellent for teaching and training undergraduate and graduate students about the latest generation neural networks with powerful learning capabilities.
More details
Other editions
Additional editions
Nan Zheng | Pinaki Mazumder
Learning in Energy-Efficient Neuromorphic Computing: Algorithm and Architecture Co-Design
E-Book
10/2019
1st Edition
Wiley-ISTE
€116.99
Available for download
Nan Zheng | Pinaki Mazumder
Learning in Energy-Efficient Neuromorphic Computing: Algorithm and Architecture Co-Design
E-Book
10/2019
1st Edition
Wiley-ISTE
€116.99
Available for download
Persons
Nan Zheng, PhD., received a B. S. degree in Information Engineering from Shanghai Jiao Tong University, Shanghai, China, in 2011, and an M. S. and PhD in Electrical Engineering from the University of Michigan, Ann Arbor, in 2014 and 2018, respectively. His research interests include low-power hardware architectures, algorithms and circuit techniques with an emphasis on machine-learning applications.
Pinaki Mazumder, PhD, is a professor in the Department of Electrical Engineering and Computer Science at The University of Michigan. His research interests include CMOS VLSI design, semiconductor memory systems, CAD tools and circuit designs for emerging technologies including quantum MOS, spintronics, spoof plasmonics, and resonant tunneling devices.
Pinaki Mazumder, PhD, is a professor in the Department of Electrical Engineering and Computer Science at The University of Michigan. His research interests include CMOS VLSI design, semiconductor memory systems, CAD tools and circuit designs for emerging technologies including quantum MOS, spintronics, spoof plasmonics, and resonant tunneling devices.
Content
Chapter 1 Overview 1
1.1 History of Neural Networks 1
1.2 Neural Networks in Software 2
1.2.1 ANN 2
1.2.2 SNN 3
1.3 Need for Neuromorphic Hardware 4
1.4 Objectives and Outlines of the Book 6
Chapter 2 Fundamentals and Learning of Artificial Neural Networks 1
2.1 Operational Principles of Artificial Neural Networks 1
2.1.1 Inference 1
2.1.2 Learning 4
2.2 Neural Network Based Machine Learning 8
2.2.1 Supervised Learning 8
2.2.2 Reinforcement Learning 11
2.2.3 Unsupervised Learning 14
2.2.4 Case Study: Action-Dependent Heuristic Dynamic Programming 16
2.3 Network Topologies 25
2.3.1 Fully-Connected Neural Networks 25
2.3.2 Convolutional Neural Networks 28
2.3.3 Recurrent Neural Networks 31
2.4 Dataset and Benchmarks 34
2.5 Deep Learning 38
2.5.1 Pre-Deep-Learning Era 38
2.5.2 The Rise of Deep Learning 38
2.5.3 Deep Learning Techniques 39
2.5.4 Deep Neural Network Examples 48
Chapter 3 Artificial Neural Networks in Hardware 1
3.1 Overview 1
3.2 General-Purpose Processors 2
3.3 Digital Accelerators 3
3.3.1 A Digital ASIC Approach 3
3.3.2 FPGA-Based Accelerators 24
3.4 Analog/Mixed-Signal Accelerators 26
3.4.1 Neural Networks in Conventional Integrated Technology 27
3.4.2 Neural Network Based on Emerging Non-Volatile Memory 34
3.4.3 Optical Accelerator 40
3.5 Case Study: An Energy-Efficient Accelerator for Adaptive Dynamic Programming 41
3.5.1 Hardware Architecture 43
3.5.2 Design Examples 50
Chapter 4 Operational Principles and Learning in SNNs 1
4.1 Spiking Neural Networks 1
4.1.1 Popular Spiking Neuron Models 1
4.1.2 Information Encoding 4
4.1.3 Spiking Neuron vs. Non-Spiking Neuron 5
4.2 Learning in Shallow SNNs 7
4.2.1 ReSuMe 8
4.2.2 Tempotron 9
4.2.3 Spike-Timing-Dependent Plasticity 11
4.2.4 Learning through Modulating Weight-Dependent STDP in Two-Layer Neural Networks 14
4.3 Learning in Deep SNNs 34
4.3.1 SpikeProp 34
4.3.2 Stack of Shallow Networks 35
4.3.3 Conversion from ANNs 37
4.3.4 Recent Advances in Backpropagation for Deep SNNs 38
4.3.5 Learning through Modulating Weight-Dependent STDP Multi-Layer Neural Networks 39
Chapter 5 Hardware Implementations of Spiking Neural Networks 1
5.1 The Need for Specialized Hardware 1
5.1.1 Address-Event Representation 1
5.1.2 Event-Driven Computation 2
5.1.3 Inference with A Progressive Precision 4
5.1.4 Hardware Considerations for Implementing the Weight-Dependent STDP Learning Rule 10
5.2 Digital SNNs 15
5.2.1 Large-Scale SNN ASICs 15
5.2.2 Small/Moderate-Scale Digital SNNs 23
5.2.3 Hardware-Friendly Reinforcement Learning in SNNs 26
5.2.4 Hardware-Friendly Supervised Learning in Multi-Layer SNNs 31
5.3 Analog/Mixed-Signal SNNs 43
5.3.1 Basic Building Blocks 43
5.3.2 Large-Scale Analog/Mixed-Signal CMOS SNNs 47
5.3.3 Other Analog/Mixed-Signal CMOS SNN ASICs 49
5.3.4 SNNs Based on Emerging Nanotechnologies 50
5.3.5 Case Study: Memristor Crossbar-based Learning in SNNs 55
Chapter 6 Conclusions 1
6.1 Outlooks 1
6.1.1 Brain-Inspired Computing 1
6.1.2 Emerging nanotechnologies 3
6.1.3 Reliable Computing with Neuromorphic Systems 4
6.1.4 Blending of ANNs and SNNs 6
6.2 Conclusions 7
Appendix
1.1 History of Neural Networks 1
1.2 Neural Networks in Software 2
1.2.1 ANN 2
1.2.2 SNN 3
1.3 Need for Neuromorphic Hardware 4
1.4 Objectives and Outlines of the Book 6
Chapter 2 Fundamentals and Learning of Artificial Neural Networks 1
2.1 Operational Principles of Artificial Neural Networks 1
2.1.1 Inference 1
2.1.2 Learning 4
2.2 Neural Network Based Machine Learning 8
2.2.1 Supervised Learning 8
2.2.2 Reinforcement Learning 11
2.2.3 Unsupervised Learning 14
2.2.4 Case Study: Action-Dependent Heuristic Dynamic Programming 16
2.3 Network Topologies 25
2.3.1 Fully-Connected Neural Networks 25
2.3.2 Convolutional Neural Networks 28
2.3.3 Recurrent Neural Networks 31
2.4 Dataset and Benchmarks 34
2.5 Deep Learning 38
2.5.1 Pre-Deep-Learning Era 38
2.5.2 The Rise of Deep Learning 38
2.5.3 Deep Learning Techniques 39
2.5.4 Deep Neural Network Examples 48
Chapter 3 Artificial Neural Networks in Hardware 1
3.1 Overview 1
3.2 General-Purpose Processors 2
3.3 Digital Accelerators 3
3.3.1 A Digital ASIC Approach 3
3.3.2 FPGA-Based Accelerators 24
3.4 Analog/Mixed-Signal Accelerators 26
3.4.1 Neural Networks in Conventional Integrated Technology 27
3.4.2 Neural Network Based on Emerging Non-Volatile Memory 34
3.4.3 Optical Accelerator 40
3.5 Case Study: An Energy-Efficient Accelerator for Adaptive Dynamic Programming 41
3.5.1 Hardware Architecture 43
3.5.2 Design Examples 50
Chapter 4 Operational Principles and Learning in SNNs 1
4.1 Spiking Neural Networks 1
4.1.1 Popular Spiking Neuron Models 1
4.1.2 Information Encoding 4
4.1.3 Spiking Neuron vs. Non-Spiking Neuron 5
4.2 Learning in Shallow SNNs 7
4.2.1 ReSuMe 8
4.2.2 Tempotron 9
4.2.3 Spike-Timing-Dependent Plasticity 11
4.2.4 Learning through Modulating Weight-Dependent STDP in Two-Layer Neural Networks 14
4.3 Learning in Deep SNNs 34
4.3.1 SpikeProp 34
4.3.2 Stack of Shallow Networks 35
4.3.3 Conversion from ANNs 37
4.3.4 Recent Advances in Backpropagation for Deep SNNs 38
4.3.5 Learning through Modulating Weight-Dependent STDP Multi-Layer Neural Networks 39
Chapter 5 Hardware Implementations of Spiking Neural Networks 1
5.1 The Need for Specialized Hardware 1
5.1.1 Address-Event Representation 1
5.1.2 Event-Driven Computation 2
5.1.3 Inference with A Progressive Precision 4
5.1.4 Hardware Considerations for Implementing the Weight-Dependent STDP Learning Rule 10
5.2 Digital SNNs 15
5.2.1 Large-Scale SNN ASICs 15
5.2.2 Small/Moderate-Scale Digital SNNs 23
5.2.3 Hardware-Friendly Reinforcement Learning in SNNs 26
5.2.4 Hardware-Friendly Supervised Learning in Multi-Layer SNNs 31
5.3 Analog/Mixed-Signal SNNs 43
5.3.1 Basic Building Blocks 43
5.3.2 Large-Scale Analog/Mixed-Signal CMOS SNNs 47
5.3.3 Other Analog/Mixed-Signal CMOS SNN ASICs 49
5.3.4 SNNs Based on Emerging Nanotechnologies 50
5.3.5 Case Study: Memristor Crossbar-based Learning in SNNs 55
Chapter 6 Conclusions 1
6.1 Outlooks 1
6.1.1 Brain-Inspired Computing 1
6.1.2 Emerging nanotechnologies 3
6.1.3 Reliable Computing with Neuromorphic Systems 4
6.1.4 Blending of ANNs and SNNs 6
6.2 Conclusions 7
Appendix