Deep Learning
Description
An engaging and accessible introduction to deep learning perfect for students and professionals
In Deep Learning: A Practical Introduction, a team of distinguished researchers delivers a book complete with coverage of the theoretical and practical elements of deep learning. The book includes extensive examples, end-of-chapter exercises, homework, exam material, and a GitHub repository containing code and data for all provided examples.
Combining contemporary deep learning theory with state-of-the-art tools, the chapters are structured to maximize accessibility for both beginning and intermediate students. The authors have included coverage of TensorFlow, Keras, and Pytorch. Readers will also find:
- Thorough introductions to deep learning and deep learning tools
- Comprehensive explorations of convolutional neural networks, including discussions of their elements, operation, training, and architectures
- Practical discussions of recurrent neural networks and non-supervised approaches to deep learning
- Fulsome treatments of generative adversarial networks as well as deep Bayesian neural networks
Perfect for undergraduate and graduate students studying computer vision, computer science, artificial intelligence, and neural networks, Deep Learning: A Practical Introduction will also benefit practitioners and researchers in the fields of deep learning and machine learning in general.
More details
Other editions
Additional editions
Persons
Manel Martínez-Ramón, PhD, is King Felipe VI Endowed Chair and Professor in the Department of Electrical and Computer Engineering at the University of New Mexico in the United States. He earned his doctorate in Telecommunication Technologies at the Universidad Carlos III de Madrid in 1999.
Meenu Ajith, PhD, is a Postdoctoral Research Associate in Tri-Institutional Center for Translational Research in Neuroimaging and Data Science (TReNDS) at Georgia State University, Georgia Institute of Technology, and Emory University. She earned her doctorate degree in Electrical Engineering from the University of New Mexico in 2022. Her research interests include machine learning, computer vision, medical imaging, and image processing.
Aswathy Rajendra Kurup, PhD, is a Data Scientist at Intel Corporation. She earned her doctorate degree in Electrical Engineering from the University of Mexico in 2022. Her research interests include image processing, signal processing, deep learning, computer vision, data analysis and data processing.
Content
About the Authors xv
Foreword xvii
Preface xix
Acknowledgment xxi
About the Companion Website xxiii
1 The Multilayer Perceptron 1
1.1 Introduction 1
1.2 The Concept of Neuron 2
1.3 Structure of a Neural Network 14
1.4 Activations 21
1.5 Training a Multilayer Perceptron 22
1.6 Conclusion 37
2 Training Practicalities 41
2.1 Introduction 41
2.2 Generalization and Overfitting 42
2.3 Regularization Techniques 45
2.4 Normalization Techniques 50
2.5 Optimizers 52
2.6 Conclusion 58
3 Deep Learning Tools 61
3.1 Python: An Overview 61
3.2 NumPy 72
3.3 Matplotlib 83
3.4 Scipy 97
3.5 Scikit-Learn 107
3.6 Pandas 116
3.7 Seaborn 125
3.8 Python Libraries for NLP 131
3.9 TensorFlow 138
3.10 Keras 141
3.11 Pytorch 144
3.12 Conclusion 149
4 Convolutional Neural Networks 153
4.1 Introduction 153
4.2 Elements of a Convolutional Neural Network 153
4.3 Training a CNN 160
4.4 Extensions of the CNN 166
4.5 Conclusion 184
5 Recurrent Neural Networks 187
5.1 Introduction 187
5.2 RNN Architecture 188
5.3 Training an RNN 191
5.4 Long-Term Dependencies: Vanishing and Exploding Gradients 199
5.5 Deep RNN 201
5.6 Bidirectional RNN 203
5.7 Long Short-Term Memory Networks 204
5.8 Gated Recurrent Units 218
5.9 Conclusion 221
6 Attention Networks and Transformers 225
6.1 Introduction 225
6.2 Attention Mechanisms 227
6.3 Transformers 242
6.4 BERT 249
6.5 GPT-2 256
6.6.1 Comparison between ViTs and CNNs 264
6.7 Conclusion 269
7 Deep Unsupervised Learning I 273
7.1 Introduction 273
7.2 Restricted Boltzmann Machines 274
7.3 Deep Belief Networks 278
7.4 Autoencoders 279
7.5 Undercomplete Autoencoder 284
7.6 Sparse Autoencoder 285
7.7 Denoising Autoencoders 287
7.8 Convolutional Autoencoder 288
7.9 Variational Autoencoders 291
7.10 Conclusion 297
8 Deep Unsupervised Learning II 301
8.1 Introduction 301
8.2 Elements of GAN 303
8.3 Training a GAN 305
8.4 Wasserstein GAN 309
8.5 DCGAN 312
8.6 cGAN 316
8.7 CycleGAN 318
8.8 StyleGAN 323
8.9 StackGAN 328
8.10 Diffusion Models 333
8.11 Conclusion 338
9 Deep Bayesian Networks 341
9.1 Introduction 341
9.2 Bayesian Models 342
9.3 Bayesian Inference Methods for Deep Learning 344
9.4 Conclusion 352
Problems 353
List of Acronyms 355
Notation 359
Bibliography 365
Index 387
1
The Multilayer Perceptron
1.1 Introduction
The concept of artificial intelligence (AI) is relatively simple to explain, and it can be enunciated as a possible answer to the question of how to make a machine that is able to perform a given task without being explicitly programmed for it, but instead, extracting the necessary information from a set of data. Let us say, for example, that a machine is needed to classify green and red apples. The machine is provided with a camera, and all the mechanisms necessary to place one apple at a time in front of it and then throw it in one of two buckets. A machine wired to do this will relay in binary operators as "IF," "THEN." If the color is red, throw it in bucket A, otherwise, in bucket "B."
The limitations of this method are obvious. If a pear is mistakenly introduced in the process, it will be classified as a green apple. Also, how can we use the same or similar structure for a different or more complex task? As in the previous machine, an AI approach uses features found in the data in order to take the decision, but the algorithm is not explicitly programmed. Instead, the machine has a specific parametric structure capable of learning from data. The learning process involves the optimization of a certain measurable criterion with respect to the parameters. The deep learning (DL) structures for artificial intelligence are able to learn complex tasks from the available data, but they also have capabilities such as learning how to extract the useful features for the task at hand, provide probabilistic outputs (i.e. "the probability of apple is 97%"), and many others. The basic element of such a structure in DL is the so-called artificial neuron, a simple concept that provides the power and nonlinear properties.
This chapter is intended to be a first contact with DL, where we introduce the most basic type of feedforward neural network (FFNN), which is called the multilayer perceptron (MLP). Here, we first introduce the low-level basic elements of most neural network (NN)s, then the structure and learning criteria.
The elements introduced in this chapter will be used throughout the book. We start from the single perceptron, we construct a basic MLP, where the different activations are developed, and then the notation based on tensors is also justified as a generalized tool to be used throughout the book. After this, we present the maximum likelihood (ML) criterion as a general criterion, which is then particularized to the classic cases corresponding to the different output activations. Finally, the backpropagation (BP) is detailed and then summarized so that can be translated into a computer program.
In this chapter, examples and exercises are presented in a way that assumes that the student does not necessarily know about programming in Python. Examples will be focused on the behavior of the MLP, without focusing on the programming, and the exercises intended to modify, at a high-level data, parameters, and structures in order to answer questions to different practical cases. Chapter 3 explains, in particular, how the different examples have been coded, thus they will be reviewed in that chapter from the point of view of practical programming.
1.2 The Concept of Neuron
The idea of the artificial neural network (ANN) is obviously inspired by the structure of the nervous system. The first attempt to understand how neural tissue works from a logical perspective was published in 1943 by Warren S. McCulloch and Walter Pitts (1943) (Fig. 1.1). They proposed the first mathematical model for a biological neuron in his paper. In this model, the neuron has two possible states, defined as 0 or 1 depending on whether the neuron is resting or it has been activated or fired. This represents the axon of the neuron. The input of this neuron model consists of a number of dendrites whose excitation is also binary. This elemental structure is completed with an inhibitory input. If this input is activated, the neuron cannot fire. If the inhibitory input is deactivated, the neuron can be activated if the combination of inputs is larger than a given threshold. This model is fully binary and, since it includes mathematical functions that cannot be differentiated, it cannot be treated mathematically in an easy way. Certain modifications that will be seen further give rise to what is known as the artificial neuron in use today.
Figure 1.1 Warren S. McCulloch (left) and Walter Pitts in 1949.
Source: R. Moreno-Díaz and A. Moreno-Díaz (2007)/with permission from Elsevier.
Section 1.2.1 contains an introduction to the concept of artificial perceptron from an algebraic point of view. A possible way to train a single perceptron is introduced in Sections 1.2.2 and 1.2.3, as well as the limitations of this structure as a linear classifier.
The concept of artificial NN was introduced by the psychologist Frank Rosenblatt (Fig. 1.2) in 1958 Rosenblatt (1957, 1958). In this paper, he proposed a structure of the visual cortex perceptron (Fig. 1.3). The structure presented in Rosenblatt (1958) contained the fundamental idea that is used in any artificial learning structure. In the first stage (Retina), the device collects the available observation or input pattern intended to be processed in order to extract knowledge of it. The second stage (Projection area) is in charge of processing this observation to extract the information needed for the task at hand. This information is commonly called the set of features of the input pattern. The third stage (Association area) is intended to process these features to map them into a given response. For example, the response may be to recognize some given object classes present in the scene. Rosenblatt is the father of the artificial perceptron. He proved that by modifying the McCulloch-Pitts neuron model, the neuron could actually learn tasks from the data. In particular, his model had weights that multiplied each of the inputs to the neuron as well as the input bias or threshold that could be adjusted for the neuron to perform a given task. He developed the Mark 1 perceptron machine, which was the first implementation of his perceptron algorithm. This device was not a computer but an electromechanical learning machine. The machine consisted of a camera constructed with an array of 400 photocells, the output of each one connected randomly to the dendrites of a set of neurons. The weights, or attenuations applied to these inputs, were controlled with potentiometers whose axes were connected to electric motors. During the learning procedure, the motors adjusted the input weights. This machine was able to distinguish linearly separable patterns, or patterns that were at one or another side of a hyperplane in the space of 400 dimensions spanned by the camera inputs depending on its binary class. The invention was then limited in its capabilities until it was proven that a perceptron constructed with more than one layer of neurons MLP had nonlinear capabilities, that is, the ability to separate patterns that could not be separated by a hyperplane. Nevertheless, the MLP could not be trained using the techniques introduced by Rosenblatt for his perceptron. It was in 1971 that Paul Werbos, in his PhD thesis (P. J. Werbos 1974) introduced the BP algorithm, which made it possible to adjust the weights of a multilayer perceptron.
Figure 1.2 Frank Rosenblatt.
Source: https://news.cornell.edu/stories/2019/09/professors-perceptron-paved-way-ai-60-years-too-soon/ last accessed November 30, 2023.
Figure 1.3 The perceptron as described in Rosenblatt (1958)/American Psychological Association.
1.2.1 The Perceptron
From a conceptual point of view, a perceptron is a function made to perform a binary classification. In order to describe this function, let us first introduce the necessary notation and concepts associated with it. Assume a given observation that consists of a collection of magnitudes observed from a physical phenomenon. These magnitudes are stored in a column vector, which will be called , which lies in a space of dimensions. For illustrative purposes, let us construct a set of artificial data in a space of dimensions as in Fig. 1.4.
The figure shows a set of points with coordinates , where operator denotes the transpose operation, meaning that the vector is a column one even if it is written as a row vector. In this toy example, the data belongs to one of two classes (black or white) that we will label arbitrarily with the labels , though in some cases, labels are more convenient. It can be seen that the data is linearly separable, that is, both classes can be separated by placing a line between the black and white clusters of data. That is, roughly speaking, the idea of the perceptron. It must be trained to place a separating hyperplane between both classes. We define the hyperplane (particularized to a line in the two-dimensional example) as
(1.1)Figure 1.4 A set of observations in a space of two dimensions.
...