Preface

In 1987, when I was wrapping up my doctoral thesis at the University of Illinois, I had a rare opportunity to listen to Prof. John Hopfield of the California Institute of Technology describing his groundbreaking research in neural networks to spellbound students in the Loomis Laboratory of Physics at Urbana-Champaign. He didactically described how to design and fabricate a recurrent neural network chip to rapidly solve the benchmark Traveling Salesman Problem (TSP), which is provably NP-complete in the sense that no physical computer could solve the problem in asymptotically bounded polynomial time as the number of cities in the TSP increases to a very large number.

This discovery of algorithmic hardware to solve intractable combinatorics problems was a major milestone in the field of neural networks as the prior art of perceptron-type feedforward neural networks could merely classify a limited set of simple patterns. Though, the founder of neural computing, Prof. Frank Rosenblatt of Cornel University had built a Mark 1 Perceptron computer in the late 1950s when the first waves of digital computers such as IBM 650 were just commercialized. Subsequent advancements in neural hardware designs were stymied mainly because of lack of integration capability of large synaptic networks by using the then technology, comprising vacuum tubes, relays, and passive components such as resistors, capacitors, and inductors. Therefore, in 1985, when AT&T Bell Labs fabricated the first solid-state proof-of-concept TSP chip by using MOS technology to verify Prof. John Hopfield's neural net architecture, it opened the vista for solving non-Boolean and brain-like computing on silicon.

Prof. John Hopfield's seminal work established that if the "objective function" of a combinatorial algorithm can be expressed in quadratic form, the synaptic links in a recurrent artificial neural network could be accordingly programmed to reduce (i.e. locally minimize) the value of the objective function through massive interactions between the constituent neurons. Hopfield's neural network consists of laterally connected neurons that can be randomly initialized and then the network can iteratively reduce the intrinsic Lyapunov energy function of the network to reach a local minima state. Notably, the Lyapunov function decreases in a monotone fashion under the dynamics of the recurrent neural networks, where neurons are not provided with self-feedback.1

Prof. Hopfield used a combination of four separate quadratic functions to represent the objective function of the TSP. The first part of the objective function ensures that the energy function minimizes if the traveling salesman traverses cities exactly once, the second part ensures that the traveling salesman visits all cities in the itinerary, the third part ensures that no two cities are visited simultaneously, and the fourth part of the quadratic function is designed to determine the shortest route connecting all cities in the TSP. Because of massive simultaneous interactions between neurons through the connecting synapses that are precisely adjusted to meet the constraints in the above quadratic functions, a simple recurrent neural network could rapidly generate a very good quality solution. However, unlike well-tested software procedures such as simulated annealing, dynamic programming, and the branch-and-bound algorithm, neural networks generally fail to find the best solution because of their simplistic connectionist structures.

Therefore, after listening to Prof. Hopfield's fascinating talk I harbored a mixed feeling about the potential benefit of his innovation. On the one hand, I was thrilled to learn from his lecture how computationally hard algorithmic problems could be solved very quickly by using simple neuromorphic CMOS circuits having very small hardware overheads. On the other hand, I thought that the TSP application that Prof. Hopfield selected to demonstrate the ability of neural networks to solve combinatorial optimization problems was not the right candidate, as software algorithms are well crafted to obtain nearly the best solution that the neural networks can hardly match. I started contemplating developing self-healing VLSI chips where the power of neural-inspired self-repair algorithms could be used to automatically restructure faulty VLSI chips. Low overheads and the ability to solve a problem concurrently through parallel interactions between neurons are two salient features that I thought could be elegantly deployed for automatically repairing VLSI chips by built-in neural net circuitry.

Soon after I joined the University of Michigan as an assistant professor, working with one of my doctoral students [2], and, at first, I developed a CMOS analog neural net circuitry with asynchronous state updates, which lacked robustness due to process variation within a die. In order to improve the reliability of the self-repair circuitry, an MS student [3] and I designed a digital neural net circuitry with synchronous state updates. These neural circuits were designed to repair VLSI chips by formulating the repair problem in terms of finding the node cover, edge cover, or node pair matching in a bipartite graph. In our graph formalism, one set of vertices in the bipartite graph represented the faulty circuit elements, and the other set of vertices represented the spare circuit elements. In order to restructure a faulty VLSI chip into a fault-free operational chip, the spare circuit elements were automatically invoked through programmable switching elements after identifying the faulty elements through embedded built-in self-testing circuitry.

Most importantly, like the TSP problem, the two-dimensional array repair can be shown to be an NP-complete problem because the repair algorithm seeks the optimal number of spare rows and spare columns that can be assigned to bypass faulty components such as memory cells, word-line and bit-line drivers, and sense amplifier bands located inside the memory array. Therefore, simple digital circuits comprising counters and other blocks woefully fail to solve such intractable self-repair problems. Notably, one cannot use external digital computers to determine how to repair embedded arrays, as input and output pins of the VLSI chip cannot be deployed to access the fault patterns in the deeply embedded arrays.

In 1989 and 1992, I received two NSF grants to expand the neuromorphic self-healing design styles to a wider class of embedded VLSI modules such as memory array [4], processors array [5], programmable logic array, and so on [6]. However, this approach to improving VLSI chip yield by built-in self-testing and self-repair was a bit ahead of its time as the state-of-the-art microprocessors in the early 1990s contained only a few hundred thousands of transistors as well as the submicron CMOS technology that was relatively robust. Therefore, after developing the neural-net based self-healing VLSI chip design methodology for various types of embedded circuit blocks, I stopped working on CMOS neural networks. I was not particularly interested in pursuing applications of neural networks for other types of engineering problems, as I wanted to remain focused on solving emerging problems in VLSI research.

On the other hand, in the late 1980s there were mounting concerns among CMOS technology prognosticators about the impending red brick wall heralding the end of the shrinking era in CMOS. Therefore, to promote several types of emerging technologies that might push the frontier of VLSI technology, the Defense Advanced Research Projects Agency (DARPA) in the USA had initiated (around 1990) the Ultra Electronics: Ultra Dense, Ultra Fast Computing Components Research Program. Concurrently, the Ministry of International Trade & Industry (MITI) in Japan had launched the Quantum Functional Devices (QFD) Project. Early successes with a plethora of innovative non-CMOS technologies in both research programs led to the launching of the National Nanotechnology Initiative (NNI), which is a U.S. Government research and development (R&D) initiative, involving 20 departments and independent agencies to bring about revolution in nanotechnology to impact the industry and society at large.

During the period of 1995 and 2010, my research group had at first focused on a quantum physics based device and circuit modeling for quantum tunneling devices, and then we extensively worked on cellular neural network (CNN) circuits for image and video processing by using one-dimensional (double barrier resonant tunneling device), two-dimensional (self-assembled nanowire), and three-dimensional (quantum dot array) constrained quantum devices. Subsequently, we developed learning-based neural network circuits by using resistive synaptic devices (commonly known as memristors) and CMOS neurons. We also developed analog voltage programmable nanocomputing architectures by hybridizing quantum tunneling and memristive devices in computing nodes of a two-dimensional processing element (PE) ensemble. Our research on nanoscale neuromorphic circuits will soon be published in our new book, titled: Neuromorphic Circuits for Nanoscale Devices, River Publishing, U.K., 2019.

After spending a little over a decade developing...