1
Deep Reinforcement Learning and Its Applications

1.1 Wireless Networks and Emerging Challenges

Over the past few years, communication technologies have been rapidly developing to support various aspects of our daily lives, from smart cities and healthcare to logistics and transportation. This will be the backbone for the future's data-centric society. Nevertheless, these new applications generate a tremendous amount of workload and require high-reliability and ultrahigh-capacity wireless communications. In the latest report [1], Cisco projected the number of connected devices that will be around 29.3 billion by 2023, with more than 45% equipped with mobile connections. The fastest-growing mobile connection type is likely machine-to-machine (M2M), as Internet-of-Things (IoT) services play a significant role in consumer and business environments. This poses several challenges in future wireless communication systems:

• Emerging services (e.g. augmented reality [AR] and virtual reality [VR]) require high-reliability and ultrahigh capacity wireless communications. However, existing communication systems, designed and optimized based on conventional communication theories, significantly prevent further performance improvements for these services.
• Wireless networks are becoming increasingly ad hoc and decentralized, in which mobile devices and sensors are required to make independent actions such as channel selections and base station associations to meet the system's requirements, e.g. energy efficiency and throughput maximization. Nonetheless, the dynamics and uncertainty of the systems prevent them from obtaining optimal decisions.
• Another crucial component of future network systems is network traffic control. Network control can dramatically improve resource usage and the efficiency of information transmission through monitoring, checking, and controlling data flows. Unfortunately, the proliferation of smart IoT devices and ultradense radio networks has greatly expanded the network size with extremely dynamic topologies. In addition, the explosive growing data traffic imposes considerable pressure on Internet management. As a result, existing network control approaches may not effectively handle these complex and dynamic networks.
• Mobile edge computing (MEC) has been recently proposed to provide computing and caching capabilities at the edge of cellular networks. In this way, popular contents can be cached at the network edge, such as base station, end-user devices, and gateways to avoid duplicate transmissions of the same content, resulting in better energy and spectrum usage [2, 3]. One major challenge in future communication systems is the straggling problems at both edge nodes and wireless links, which can significantly increase the computation delay of the system. Additionally, the huge data demands of mobile users and the limited storage and processing capacities are critical issues that need to be addressed.

Conventional approaches to addressing the new challenges and demands of modern communication systems have several limitations. First, the rapid growth in the number of devices, the expansion of network scale, and the diversity of services in the new era of communications are expected to significantly increase the amount of data generated by applications, users, and networks [1]. However, traditional solutions may be unable to process and utilize this data effectively to improve system performance. Second, existing algorithms are not well-suited to handle the dynamic and uncertain nature of network environments, resulting in poor performance [4]. Finally, traditional optimization solutions often require complete information about the system to be effective, but this information may not be readily available in practice, limiting the applicability of these approaches. Deep reinforcement learning (DRL) has the potential to overcome these limitations and provide promising solutions to these challenges.

DRL leverages the benefits of deep neural networks (DNNs), which have proven effective in tackling complex, large-scale engines, speech recognition, medical diagnosis, and computer vision. This makes DRL well suited for managing the increasing complexity and scale of future communication networks. Additionally, DRL's online deployment allows it to effectively handle the dynamics and unpredictable nature of wireless communication environments.

1.2 Machine Learning Techniques and Development of DRL

1.2.1 Machine Learning

Machine learning (ML) is a problem-solving paradigm where a machine learns a particular task (e.g. image classification, document text classification, speech recognition, medical diagnosis, robot control, and resource allocation in communication networks) and performance metric (e.g. classification accuracy and performance loss) using experiences or data [5]. The task generally involves a function that maps well-defined inputs to well-defined outputs. The essence of data-driven ML is that there is a pattern in the task inputs and the outcome which cannot be pinned down mathematically. Thus, the solution to the task, which may involve making a decision or predicting an output, cannot be programmed explicitly. If the set of rules connecting the task inputs and output(s) were known, a program could be written based on those rules (e.g. if-then-else codes) to solve the problem. Instead, an ML algorithm learns from the input data set, which specifies the correct output for a given input; that is, an ML method will result in a program that uses the data samples to solve the problem. A data-driven ML architecture for the classification problem is shown in Figure 1.1. The training module is responsible for optimizing the classifier from the training data samples and providing the classification module with a trained classifier. The classification module determines the output based on the input data. The training and classification modules can work independently. The training procedure generally takes a long time. However, the training module is activated only periodically. Also, the training procedure can be performed in the background, while the classification module operates as usual.

Figure 1.1 A data-driven ML architecture.

There are three categories of ML techniques, including supervised, unsupervised, and reinforcement learning.

• Supervised learning: Given a data set , a supervised learning algorithm predicts that generalizes the input-output mapping in to inputs outside . Here, is the -dimensional feature space , is the input vector of the th sample, is the label of the th sample, and is the label space. For binary classification problems (e.g. spam filtering), or . For multiclass classification (e.g. face classification), . On the other hand, for regression problems (e.g. predicting temperature), . The data points are drawn from a (unknown) distribution . The learning process involves learning a function such that for a new pair , we have with high probability (or ). A loss function (or risk function), such as the mean squared error function, evaluates the error between the predicted probabilities/values returned by the function and the labels on the training data.

  For supervised learning, the data set is usually split into three subsets: as the training data, as the validation data, and as the test data. The function is validated on : if the loss is too significant, will be revised based on and validated again on . This process will keep going back and forth until it gives a low loss on . The standard supervised learning techniques include the following: Bayesian classification, logistic regression, -nearest neighbor (KNN), neural network (NN), support vector machine (SVM), decision tree (DT) classification, and recommender system. Note that supervised learning techniques require the availability of labeled data sets.

• Unsupervised learning techniques are used to create an internal representation of the input, e.g. to form clusters, extract features, reduce dimensionality, estimate density. Unlike supervised learning, these techniques can deal with unlabeled data sets.
• Reinforcement learning (RL) techniques do not require a prior dataset. With RL, an agent learns from interactions with an external environment. The idea of learning by interacting with a domain is an imitation of humans' natural learning process. For example, at the point when a newborn child plays, e.g. waves his arms or kicks a ball, his/her brain has a direct sensorimotor connection with its surroundings. Repeating this process produces essential information about the impact of actions, causes and effects, and what to do to reach the goals.

Deep learning (DL), a subset of ML, has gained popularity thanks to its DNN architectures to overcome the limitations of ML. DL models are able to extract the key features of data without relying on the data's structure. The "deep" in deep learning refers to the number of layers in the DNN architecture, with more layers leading to a deeper network. DL has been successfully applied in various fields, including face and voice recognition, text translation, and intelligent driver assistance systems. It has several advantages over traditional algorithms as follows...