1
Machine Learning: An Introduction to Reinforcement Learning

Sheikh Amir Fayaz1, Dr. S Jahangeer Sidiq2*, Dr. Majid Zaman3 and Dr. Muheet Ahmed Butt1

1Department of Computer Science University of Kashmir, Srinagar, J&K, India

2School of Computer Applications Lovely Professional University, Phagwara, Punjab, India

3Directorate of IT&SS University of Kashmir, Srinagar, J&K, India

Abstract

Reinforcement Learning (RL) is a prevalent prototype for finite sequential decision making under improbability. A distinctive RL algorithm functions with only restricted knowledge of the environment and with limited response or feedback on the quality of the conclusions. To work efficiently in complex environments, learning agents entail the capability to form convenient generalizations, that is, the ability to selectively overlook extraneous facts. It is a challenging task to develop a single illustration that is suitable for a large problem setting. This chapter provides a brief introduction to reinforcement learning models, procedures, techniques, and reinforcement learning processes. Particular focus is on the aspects related to the agent-environment interface and how Reinforcement Learning can be used in various daily life practical applications. The basic concept that we will explore is that of a solution to the Re-enforcement Learning problem using the Markov Decision Process (MDP). We assume that the reader has a basic idea of machine learning concepts and algorithms.

Keywords: Reinforcement learning, Markov decision process, agent-environment interaction, exploitation, exploration

1.1 Introduction

1.1.1 Motivation

Many years ago, we started automating physical solutions with machines during the Industrial Revolution, replacing animals with machines. We kind of know how to pull something forward across a track and then we just implement those machines. We use the machines instead of humans and animals as laborers and, of course, if we want to make a huge boom in productivity, then after that, the second wave of automation occurred and is basically still happening. Now, we are in you could call the Digital Revolution, where instead of taking physical solutions, we took mental solutions as a canonical example. For example, if we have a calculator and we know how to do division, we can program that into a calculator and have the machine do the mental task in future operations. Though we automated its mental solutions, we still come up with the solutions by ourselves, i.e., we came up with what we want to do and how to do it and then we implemented it on the machine. The next step here is to define the problem, then have a machine solve for it itself. For this, we require learning, i.e., we require something in addition because if we do not put anything into the system, how can we know it. One thing we can put into a system is our knowledge. This is what was done with these machines either for mental or physical solutions, but the other thing we could put in there is some knowledge on how the machine could learn itself.

This chapter is structured as follows: the current section provides a brief introduction about machine learning and its types. The main focus in Section 1.1.3.3 has been put on Reinforcement Learning and its types, learning process, and RL learning concepts. Section 1.2 introduces a Reinforcement Learning Paradigm and its characteristics. Section 1.3 defines the Reinforcement Learning (RL) problem where the goal is to select the actions to make the most of the total rewards. Section 1.4 gives the basic algorithm which defines the solution of the reinforcement problem using the Markov Decision Process (MDP) with its mathematical Notation. Lastly, Section 1.5 briefly explains the applications of the Reinforcement Learning process in the current day to day learning environment.

1.1.2 Machine Learning

Machine learning is the science of getting computers to act by feeding them data and letting them learn a few trails on their own. Here, we are not actually programming the machine but feeding the machine lots of data so that it can learn by itself [15, 20].

Definition: Machine learning is as subset of Artificial Intelligence (AI) which provides machines the capability to learn automatically and improve from experience without being explicitly programmed.

Example:

As kids we would not be able to differentiate between fruits like apples, cherries, and oranges. When we first see how apples, oranges, and cherries look, we were not able to differentiate between them. This is because we have not observed them enough from what they look like, but as we grow up, we learn what apples, cherries, and oranges look like, i.e., we came to know the color, shape, and size of an apple and so on. Similarly, when it comes to a machine, if we input images of an apple or any fruit, initially it will not be able to differentiate between them. This is because it does not have enough data about them, but if we keep feeding the machine with a number of images about these (cherries, apples, and oranges), it will learn how to differentiate between the three. Just like humans learn by observing and collecting data, similarly machines also learn when we give a lot of data, i.e., when we input a lot of data to a machine it will learn how to distinguish between them. The machine will basically train in the data and try to differentiate between the various rules by using various machine learning algorithms.

1.1.3 How Machines Learn

There are three different types of machine learning:

1. 1) Supervised Learning
2. 2) Unsupervised Learning
3. 3) Reinforcement Learning

1.1.3.1 Supervised Learning

This type of learning means to direct a certain activity and make sure it is done correctly. This is one of the popular machine learning methods. Several classification and regression applications were developed recently by researchers [52-58]. If we feed the model with a set of data, called training data, which contains both the inputs and the corresponding expected output, then the training data will teach the model for the correct output for a particular input [14, 17-19]. The data in supervised learning is labeled data. In supervised learning we have a joint distribution of data points and labels [47-49]. Our goal is to maximize the probability of the classifier as shown below:

where "Ds" is the labeled data, and x (input) and y (output) are the samples from p(x,y), which is a marginal response to the joint distribution from the labelled data set sample.

1.1.3.2 Unsupervised Learning

This type of learning acts without anyone's direction or supervision. In this method, the model is basically given a dataset which is neither labeled nor classified [22, 23], so the machine does not know how the output should look [31, 50, 51]. For example, if we input an image of a fruit, we are not going to label it as say "apple". Instead, we only give data and the machine is going to learn on its own.

where "Du" is the unlabeled data and x (input) is the sample from p(x).

1.1.3.3 Reinforcement Learning

Is one of the types of machine learning which has been growing in the past few years. It has various applications like self-driving or automated cars and image processing techniques wherein we detect objects using reinforcement learning. This type of machine learning has become one of the most important types in the current world to establish or to encourage a pattern of behavior. It is a learning technique where an agent interrelates through its environment by generating actions and discovering errors or rewards. In other words, Reinforcement Learning is a type of a machine learning where an agent learns to act in an environment by carrying out actions and observing the results [1, 16].

Reinforcement Learning is all about taking a suitable action in order to make the most of the reward in a particular state or situation. In this learning there is no predictable output as in supervised learning; here, the reinforcement agent chooses which action to take in order to perform a specified task. In the absence of a training dataset, it is bound to acquire from its experience itself [2, 3].

Thus, Reinforcement Learning is the type of machine learning where machines learn from evaluation, whereas other learning algorithms (supervised learning) learn from instructions. So, Reinforcement Learning is essentially an interaction between the agent and the environment (Figure 1.1) in the fashion that the agent performs an action which makes some changes in the environment and based on that, the environment gives a reward (positive or negative) to the agent. If the reward is positive, that reinforces that particular action and if the award is negative that tends to prevent the agent from performing that same action again [21]. So, this process goes on for over many iterations until the agent has learned to interact with an environment in a meaningful way. This is an essence of reinforcement learning.

Let us try to understand the concept of reinforcement learning through an example of Tic-Tac-Toe which you are familiar with and might have played a lot in your childhood. This game (Figure 1.2) essentially consists of 9 places on the square board and you need to populate the 9 places with two...