Preface
Deep Learning with TensorFlow and Keras, Third Edition, is a concise yet thorough introduction to modern neural networks, artificial intelligence, and deep learning technologies designed especially for software engineers and data scientists. The book is the natural follow-up of the books Deep Learning with Keras [1] and TensorFlow 1.x Deep Learning Cookbook [2] previously written by the same authors.
This book provides a very detailed panorama of the evolution of learning technologies over the past six years. The book presents dozens of working deep neural networks coded in Python using TensorFlow 2.x, a modular network library based on Keras-like APIs [1].
Artificial Intelligence (AI) lays the ground for everything this book discusses. Machine Learning (ML) is a branch of AI, and Deep Learning (DL) is in turn a subset of ML. This section will briefly discuss these three concepts, which you will regularly encounter throughout the rest of this book.
AI denotes any activity where machines mimic intelligent behaviors typically shown by humans. More formally, it is a research field in which machines aim to replicate cognitive capabilities such as learning behaviors, proactive interaction with the environment, inference and deduction, computer vision, speech recognition, problem-solving, knowledge representation, and perception. AI builds on elements of computer science, mathematics, and statistics, as well as psychology and other sciences studying human behaviors. There are multiple strategies for building AI. During the 1970s and 1980s, "expert" systems became extremely popular. The goal of these systems was to solve complex problems by representing the knowledge with a large number of manually defined if-then rules. This approach worked for small problems on very specific domains, but it was not able to scale up for larger problems and multiple domains. Later, AI focused more and more on methods based on statistical methods that are part of ML.
ML is a subdiscipline of AI that focuses on teaching computers how to learn without the need to be programmed for specific tasks. The key idea behind ML is that it is possible to create algorithms that learn from, and make predictions on, data. There are three different broad categories of ML:
- Supervised learning, in which the machine is presented with input data and the desired output, and the goal is to learn from those training examples in such a way that meaningful predictions can be made for data that the machine has never observed before.
- Unsupervised learning, in which the machine is presented with input data only, and the machine has to subsequently find some meaningful structure by itself, with no external supervision or input.
- Reinforcement learning, in which the machine acts as an agent, interacting with the environment. The machine is provided with "rewards" for behaving in a desired manner, and "penalties" for behaving in an undesired manner. The machine attempts to maximize rewards by learning to develop its behavior accordingly.
DL took the world by storm in 2012. During that year, the ImageNet 2012 challenge was launched with the goal of predicting the content of photographs using a subset of a large hand-labeled dataset. A deep learning model named AlexNet achieved a top-5 error rate of 15.3%, a significant improvement with respect to previous state-of-the-art results. According to the Economist, Suddenly people started to pay attention, not just within the AI community but across the technology industry as a whole.
That was only the beginning. Today, DL techniques are successfully applied in heterogeneous domains including, but not limited to, healthcare, environment, green energy, computer vision, text analysis, multimedia, finance, retail, gaming, simulation, industry, robotics, and self-driving cars. In each of these domains, DL techniques can solve problems with a level of accuracy that was not possible using previous methods.
Looking back at the past eight years, it is fascinating and exciting to see the extent of the contributions that DL has made to science and industry. There is no reason to believe that the next eight years will see any less contribution; indeed, as the field of DL continues to advance, we anticipate that we'll see even more exciting and fascinating contributions provided by DL.
This book introduces you to the magic of deep learning. We will start with simple models and progressively will introduce increasingly sophisticated models. The approach will always be hands-on, with a healthy dose of code to work with.
Who this book is for
If you are a data scientist with experience in ML or an AI programmer with some exposure to neural networks, you will find this book a useful entry point to DL with TensorFlow. If you are a software engineer with a growing interest in the DL tsunami, you will find this book a foundational platform to broaden your knowledge on the topic. Basic knowledge of Python is required for this book.
What this book covers
Chapter 1, Neural Network Foundations with TF, is where we learn the basics of TensorFlow, an open-source library developed by Google for machine learning and deep learning. In addition, we introduce the basics of neural networks and deep learning, two areas of machine learning that had incredible growth during the last few years. The idea behind this chapter is to provide all the tools needed to do basic but fully hands-on deep learning.
Chapter 2, Regression and Classification, focuses on the fundamental tasks in ML techniques: regression and classification. We will learn how to use TensorFlow to build simple, multiple, and multivariate regression models. We will use logistic regression to solve a multi-class classification problem.
Chapter 3, Convolutional Neural Networks, covers how to use deep learning ConvNets for recognizing MNIST handwritten characters with high accuracy. We use the CIFAR 10 dataset to build a deep learning classifier with 10 categories, and the ImageNet dataset to build an accurate classifier with 1,000 categories. In addition, we investigate how to use large deep learning networks such as VGG16 and very deep networks such as InceptionV3. We will conclude with a discussion on transfer learning
Chapter 4, Word Embeddings, is where we describe the origins of and theory behind distributed representations and word embeddings and chart the progress of word embeddings from static word-based embeddings more dynamic and expressive embeddings based on sentences and paragraphs. We also explore how the idea of word embeddings can be extended to include non-word sequences as well, such as nodes in a graph or user sessions in a web application. The chapter also contains multiple examples of using word embeddings of various kinds.
Chapter 5, Recurrent Neural Networks, describes an important architectural subclass of neural networks that are optimized for handling sequence data such as natural language or time series. We describe the important architectures in this genre, such as LSTM (Long Short-Term Memory) and GRU (Gated Recurrent Unit) and show how they can be extended to handle bidirectional states and states across batches. We also provide examples of using RNNs with various topologies for specific tasks, such as generating text, sentiment analysis, and part-of-speech tagging. We also describe the popular seq2seq architecture, which uses a pair of RNNs in an encoder-decoder pipeline to solve a variety of NLP tasks.
Chapter 6, Transformers, covers transformers, a deep learning architecture that has revolutionized the traditional natural language processing field. We start by reviewing the key intuitions behind the architecture and various categories of transformers, together with a deep dive into the most popular models. Then, we focus on implementations both based on the vanilla architecture and on popular libraries, such as Hugging Face and TensorFlow Hub. After that, we briefly discuss evaluation, optimization, and some of the best practices commonly adopted when using transformers. The last section is devoted to reviewing how transformers can be used to perform computer vision tasks, a totally different domain from NLP. That requires a careful definition of the attention mechanism. In the end, attention is all you need! And at the core of attention, there is nothing more than the cosine similarity between vectors.
Chapter 7, Unsupervised Learning, delves into unsupervised learning models. It will cover techniques required for clustering and dimensionality reduction like PCA, k-means, and self-organized maps. It will go into the details of Boltzmann machines and their implementation using TensorFlow. The concepts covered will be extended to build Restricted Boltzmann Machines (RBMs).
Chapter 8, Autoencoders, describes autoencoders, a class of neural networks that attempt to recreate the input as its target. It will cover different varieties of autoencoders like sparse autoencoders, convolutional autoencoders, and denoising autoencoders. The chapter will train a denoising autoencoder to remove noise from input images. It will demonstrate how autoencoders can be used to create MNIST digits. It will also cover the steps...
