Introduction

I.1 Background and Motivation

I.1.1 Lack of an Efficient General Nonlinear Optimal Control Technique

Optimal control theory plays an important role in designing effective control systems. For linear systems, a class of optimal control problems are solved successfully under the framework of Linear Quadratic Regulator (LQR). LQR problems are concerned with minimizing a quadratic cost for linear systems in terms of the control input and state, solving which allows us to regulate the state and the control input of the system. In control systems applications, this provides an opportunity to specifically regulate the behavior of the system by adjusting the weighting coefficients used in the cost functional. However, when it turns to nonlinear dynamical systems, there is no systematic method for efficiently obtaining an optimal feedback control for the general nonlinear systems. Thus, many of the techniques available in the literature on linear systems do not apply in general.

Despite the complexity of nonlinear dynamical systems, they have attracted much attention from researchers in recent years. This is mostly because of their practical benefits in establishing a wide variety of applications in engineering, including power electronics, flight control, and robotics, among many others. Considering the control of a general nonlinear dynamical system, optimal control involves finding a control input that minimizes a cost functional that depends on the controlled state trajectory and the control input. While such a problem formulation can cover a wide range of applications, how to efficiently solve such problems remains a topic of active research.

I.1.2 Importance of an Optimal Feedback Control

In general, there exist two well-known approaches to solving such optimal control problems: the maximum (or minimum) principles [Pontryagin, 1987] and the Dynamic Programming (DP) method [Bellman and Dreyfus, 1962]. To solve an optimization problem that involves dynamics, maximum principles require us to solve a two-point boundary value problem, where the solution is not in a feedback form.

There exist plenty of numerical techniques presented in the literature to solve the optimal control problem. Such approaches generally rely on knowledge of the exact model of the system. In the case where such a model exists, the optimal control input is obtained in the open-loop form as a time-dependent signal. Consequently, implementing these approaches in real-world problems often involves many complications that are well known by the control community. This is because of the model mismatch, noises, and disturbances that greatly affect the online solution, causing it to diverge from the preplanned offline solution. Therefore, obtaining a closed-loop solution for the optimal control problem is often preferred in such applications.

The DP approach analytically results in a feedback control for linear systems with a quadratic cost. Moreover, employing the Hamilton-Jacobi-Bellman (HJB) equation with a value function, one might manage to derive an optimal feedback control rule for some real-world applications, provided that the value function can be updated in an efficient manner. This motivates us to consider conditions leading to an optimal feedback control rule that can be efficiently implemented in real-world problems.

I.1.3 Limits of Optimal Feedback Control Techniques

Consider an optimal control problem over an infinite horizon involving a nonquadratic performance measure. Using the idea of inverse optimal control, the cost functional can be then be evaluated in closed form as long as the running cost depends somehow on an underlying Lyapunov function by which the asymptotic stability of the nonlinear closed-loop system is guaranteed. Then it can be obtained that the Lyapunov function is indeed the solution of the steady-state HJB equation. Although such a formulation allows analytically obtaining an optimal feedback rule, choosing the proper performance measure may not be trivial. Moreover, from a practical point of view, because of the nonlinearity in the performance measure, it might cause unpredictable behavior.

A well-studied method for solving an optimal control problem online is employing a value function assuming a given policy. Then, for any state, the value function gives a measure of how good the state is by collecting the cost starting from that state while the policy is applied. If such a value function can be obtained, and the system model is known, the optimal policy is actually the one that takes the system in the direction by which the value decreases the most in the space of the states. Such Reinforcement Learning (RL) techniques, which are known as value-based methods, including the Value Iteration (VI) and the Policy Iteration (PI) algorithms, are shown to be effective in finite state and control spaces. However, the computations cannot efficiently scale with the size of the state and control spaces.

I.1.4 Complexity of Approximate DP Algorithms

One way of facilitating the computations regarding the value updates is employing an approximate scheme. This is done by parameterizing the value function and adjusting the parameters in the training process. Then, the optimal policy given by the value function is also parameterized and approximated accordingly. The complexity of any value update depends directly on the number of parameters employed, where one may try limiting the number of the parameters by sacrificing the optimality. Therefore, we are motivated to obtain a more efficient update rule for the value parameters, rather than limiting the number of the parameters. We achieve this by reformulating the problem with a quadratically parameterized value function.

Moreover, the classical VI algorithm does not explicitly use the system model for evaluating the policy. This benefits applications in that the full knowledge of the system dynamics is no longer required. However, online training with VI alone may take much longer time to converge, since the model only participates implicitly through the future state. Therefore, the learning process can be potentially accelerated by introducing the system model. Furthermore, this creates an opportunity for running a separate identifier unit, where the model obtained can be simulated offline to complete the training or can be used for learning optimal policies for different objectives.

It can be shown that the VI algorithm for linear systems results in a Lyapunov recursion in the policy evaluation step. Such a Lyapunov equation in terms of the system matrices can be efficiently solved. However, for the general nonlinear case, methods for obtaining an equivalent are not amenable to efficient solutions. Hence, we are motivated to investigate the possibility of acquiring an efficient update rule for nonlinear systems.

I.1.5 Importance of Learning-based Tracking Approaches

One of the most common problems in the control of dynamical systems is to track a desired reference trajectory, which is found in a variety of real-world applications. However, designing an efficient tracking controller using conventional methods often necessitates a thorough understanding of the model, as well as computations and considerations for each application. RL approaches, on the other hand, propose a more flexible framework that requires less information about the system dynamics. While this may create additional problems, such as safety or computing limits, there are already effective outcomes from the use of such approaches in real-world situations. Similar to regulation problems, the applications of tracking control can benefit from Model-based Reinforcement Learning (MBRL) that can handle the parameter updates more efficiently.

I.1.6 Opportunities for Obtaining a Real-time Control

In the approximate optimal control technique, employing a limited number of parameters can only yield a local approximation of the model and the value function. However, if an approximation within a larger domain is intended, a considerably higher number of parameters may be needed. As a result, the identification and the controller's complexity might be rather too high to be performed online in real-world applications. This convinces us to circumvent this constraint by considering a set of local simple learners instead, in a piecewise approach.

As mentioned, there exist already interesting real-world applications of MBRL. Motivated by this, in this monograph, we aim on introducing automated ways of solving optimal control problems that can replace the conventional controllers. Hence, detailed applications of the proposed approaches are included, which are demonstrated with numerical simulations.

I.1.7 Summary

The main motivation for this monograph can be summarized as follows:

• Optimal control is highly favored, while there is no general analytical technique applicable to all nonlinear systems.
• Feedback control techniques are known to be more robust and computationally efficient compared to the numerical techniques, especially in the continuous space.
• The chance of obtaining a feedback control in closed form is low, and the known techniques are limited to...