An Introduction To Optimal Control
An Introduction to Optimal ControlUgo BoscainBenetto PiccoliThe aim of these notes is to give an introduction to the Theory of Optimal Controlfor finite dimensional systems and in particular to the use of the Pontryagin MaximumPrinciple towards the construction of an Optimal Synthesis. In Section 1, we introducethe definition of Optimal Control problem and give a simple example. In Section 2 werecall some basics of geometric control theory as vector fields, Lie bracket and controllability. In Section 3, that is the core of these notes, we introduce Optimal Controlas a generalization of Calculus of Variations and we discuss why, if we try to writethe problem in Hamiltonian form, the dynamics makes the Legendre transformationnot well defined in general. Then we briefly introduce the problem of existence ofminimizers and state the Pontryagin Maximum Principle. As an application we consider a classical problem of Calculus of Variations and show how to derive the EulerLagrange equations and the Weierstraß condition. Then we discuss the difficulties infinding a complete solution to an Optimal Control Problem and how to attack it withgeometric methods. In Section 4 we give a brief introduction to the theory of TimeOptimal Synthesis on two dimensional manifolds developed in [14]. We end with abibliographical note and some exercises.1IntroductionControl Theory deals with systems that can be controlled, i.e. whose evolution canbe influenced by some external agent. Here we consider control systems that can bedefined as a system of differential equations depending on some parameters u U Rm :ẋ f (x, u),(1)where x belongs to some n–dimensional smooth manifold or, in particular, to R n . Foreach initial point x0 there are many trajectories depending on the choice of the controlparameters u.One usually distinguishes two different ways of choosing the control: open loop. Choose u as function of time t, closed loop or Feedback. Choose u as function of space variable x.19
20U. B OSCAINANDB. P ICCOLIThe first problem one faces is the study of the set of points that can be reached,from x0 , using open loop controls. This is also known as the controllability problem.If controllability to a final point x1 is granted, one can try to reach x1 minimizingsome cost, thus defining an Optimal Control Problem:Z TminL(x(t), u(t)) dt,x(0) x0 , x(T ) x1 ,(2)0where L : R U R is the Lagrangian or running cost. To have a precise definitionof the Optimal Control Problem one should specify further: the time T fixed or free,the set of admissible controls and admissible trajectories, etc. Moreover one can fixan initial (and/or a final) set, instead than the point x0 (and x1 ).Fixing the initial point x0 and letting the final condition x1 vary in some domain ofRn , we get a family of Optimal Control Problems. Similarly we can fix x1 and let x0vary. One main issue is to introduce a concept of solution for this family of problemsand we choose that of Optimal Synthesis. Roughly speaking, an Optimal Synthesis isa collection of optimal trajectories starting from x0 , one for each final condition x1 .As explained later, building an Optimal Synthesis is in general extremely difficult, butgeometric techniques provide a systematic method to attack the problem.In Section 3.1 Optimal Control is presented as a generalization of Calculus ofVariations subjects to nonholonomic constraints.nExample Assume to have a point of unitary mass moving on a one dimensional lineand to control an external bounded force. We get the control system:ẍ u,x R, u C,where x is the position of the point, u is the control and C is a given positive constant.Setting x1 x, x2 ẋ and, for simplicity, C 1, in the phase space the system iswritten as: ẋ1 x2ẋ2 u.One simple problem is to drive the point to the origin with zero velocity in minimumtime. From an initial position (x̄1 , x̄2 ) it is quite easy to see that the optimal strategy isto accelerate towards the origin with maximum force on some interval [0, t] and then todecelerate with maximum force to reach the origin at velocity zero. The set of optimaltrajectories is depicted in Figure 1.A: this is the simplest example of Optimal Synthesisfor two dimensional systems. Notice that this set of trajectories can be obtained usingthe following feedback, see Figure 1.B. Define the curves ζ {(x1 , x2 ) : x2 0, x1 x22 } and let ζ be defined as the union ζ {0}. We define A to be theregion below ζ and A the one above. Then the feedback is given by: 1 if (x1 , x2 ) A ζ 1 if (x1 , x2 ) A ζ u(x) 0if (x1 , x2 ) (0, 0).
A N I NTRODUCTIONTOO PTIMAL C ONTROL21x2x2ζx1 (u 1)u(x) 1x1u(x) 1(u AB1)ζ Figure 1: The simplest example of Optimal Synthesis and corresponding feedback.Notice that the feedback u is discontinuous.2Basic Facts on Geometric controlThis Section provides some basic facts about Geometric Control Theory. This is abrief introduction that is far from being complete: we illustrate some of the mainavailable results of the theory, with few sketches of proofs. For a more detailed treatment of the subject, we refer the reader to the monographs [3, 29].Consider a control system of type (1), where x takes values on some manifold Mand u U . Along these notes, to have simplified statements and proofs, we assumemore regularity on M and U :(H0) M is a closed n-dimensional submanifold of RN for some N n. The setU is a measurable subset of Rm and f is continuous, smooth with respect to xwith Jacobian, with respect to x, continuous in both variables on every chart ofM.A point of view, very useful in geometric control, is to think a control system as afamily of assigned vector fields on a manifold:F {Fu (·) f (·, u)}u U .We always consider smooth vector fields, on a smooth manifold M , i.e. smooth mappings F : x M 7 F (x) Tx M , where Tx M is the tangent space to M at x. A
22U. B OSCAINANDB. P ICCOLIvector field can be seen as an operator from the set of smooth functions on M to R. Ifx (x1 , ., xn ) is a local system of coordinates, we have:F (x) nXFii 1 . xiThe first definition we need is of the concept of control and of trajectory of a controlsystem.Definition 1 A control is a bounded measurable function u(·) : [a, b] U . A trajectory of (1) corresponding to u(·) is a map γ(·) : [a, b] M , Lipschitz continuous onevery chart, such that (1) is satisfied for almost every t [a, b]. We write Dom(γ),Supp(γ) to indicate respectively the domain and the support of γ(·). The initial pointof γ is denoted by In(γ) γ(a), while its terminal point T erm(γ) γ(b)Then we need the notion of reachable set from a point x0 M .Definition 2 We call reachable set within time T 0 the following set:Rx0 (T ) : {x M : there exists t [0, T ] and a trajectoryγ : [0, t] M of (1) such that γ(0) x0 , γ(t) x}.(3)Computing the reachable set of a control system of the type (1) is one of the mainissues of control theory. In particular the problem of proving that R x0 ( ) coincidewith the whole space is the so called controllability problem. The corresponding localproperty is formulated as:Definition 3 (Local Controllability) A control system is said to be locally controllable at x0 if for every T 0 the set Rx0 (T ) is a neighborhood of x0 .Various results were proved about controllability and local controllability. We onlyrecall some definitions and theorems used in the sequel.Most of the information about controllability is contained in the structure of theLie algebra generated by the family of vector fields. We start giving the definition ofLie bracket of two vector fields.Definition 4 (Lie Bracket) Given two smooth vector fields X, Y on a smooth manifold M , the Lie bracket is the vector field given by:[X, Y ](f ) : X(Y (f )) Y (X(f )).In local coordinates:[X, Y ]j X Yji xiXi In matrix notation, defining Y : Yj / xi XjYi . xi(j,i)(j row, i column) and thinking toa vector field as a column vector we have [X, Y ] Y · X X · Y .
A N I NTRODUCTIONTOO PTIMAL C ONTROL23Definition 5 (Lie Algebra of F) Let F be a family of smooth vector fields on asmooth manifold M and denote by χ(M ) the set of all C vector fields on M . The Liealgebra Lie(F) generated by F is the smallest Lie subalgebra of χ(M ) containingF. Moreover for every x M we define:Liex(F) : {X(x) : X Lie(F)}.(4)Remark 1 In general Lie(F) is a infinite-dimensional subspace of χ(M ). On theother side since all X(x) Tx M (in formula (4)) we have that Liex (F) Tx M andhence Liex(F) is finite dimensional.Remark 2 Lie(F) is built in the following way. Define: D1 Span{F}, D2 Span{D1 [D1 , D1 ]}, · · · Dk Span{Dk 1 [Dk 1 , Dk 1 ]}. D1 is the so calleddistribution generated by F and we have Lie(F) k 1 Dk . Notice that Dk 1 Dk . Moreover if [Dn , Dn ] Dn for some n, then Dk Dn for every k n.A very important class of families of vector fields are the so called Lie bracketgenerating (or completely nonholonomic) systems for which:LiexF Tx M, x M.(5)For instance analytic systems (i.e. with M and F analytic) are always Lie bracketgenerating on a suitable immersed analytic submanifold of M (the so called orbit ofF). This is the well know Hermann-Nagano Theorem (see for instance [29], pp. 48).If the system is symmetric, that is F F (i.e. f F f F), then thecontrollability problem is more simple. For instance condition (5) with M connectedimplies complete controllability i.e. for each x0 M , Rx0 ( ) M (this is acorollary of the well know Chow Theorem, see for instance [3]).On the other side, if the system is not symmetric (as for the problem treated inSection 4), the controllability problem is more complicated and controllability is notguaranteed in general (by (5) or other simple conditions), neither locally. Anyway,important properties of the reachable set for Lie bracket generating systems are givenby the following theorem (see [37] and [3]):Theorem 1 (Krener) Let F be a family of smooth vector fields on a smooth manifold M . If F is Lie bracket generating, then, for every T ]0, ], Rx0 (T ) Clos(Int(Rx0 (T )). Here Clos(·) and Int(·) are taken with respect to the topologyof M .Krener Theorem implies that the reachable set for Lie bracket generating systems hasthe following properties: It has nonempty interior: Int(Rx0 (T )) 6 , T ]0, ]. Typically it is a manifold with or without boundary of full dimension. Theboundary may be not smooth, e.g. have corners or cuspidal points.
24U. B OSCAINANDB. P ICCOLIFigure 2: A prohibited reachable set for a Lie bracket generating systems.In particular it is prohibited that reachable sets are collections of sets of differentdimensions as in Figure 2. These phenomena happen for non Lie bracket generatingsystems, and it is not know if reachable sets may fail to be stratified sets (for genericsmooth systems) see [28, 29, 54].Local controllability can be detached by linearization as shown by the followingimportant result (see [40], p. 366):Theorem 2 Consider the control system ẋ f (x, u) where x belongs to a smoothmanifold M of dimension n and let u U where U is a subset of Rm for some m,containing an open neighborhood of u0 Rm . Assume f of class C 1 with respect tox and u. If the following holds:f (x0 , u0 ) 0,rank[B, AB, A2 B, ., An 1 B] n,where A f / x)(x0 , u0 ) and B f / u)(x0 , u0 ),(6)then the system is locally controllable at x0 .Remark 3 Condition (6) is the well know Kalman condition that is a necessary andsufficient condition for (global) controllability of linear systems:ẋ Ax Bu, x Rn , A Rn n , B Rn m , u Rm .In the local controllable case we get this further property of reachable sets:Lemma 1 Consider the control system ẋ f (x, u) where x belongs to a smoothmanifold M of dimension n and let u U where U is a subset of R m for some m.
A N I NTRODUCTIONTOO PTIMAL C ONTROL25Assume f of class C 1 with respect to x and continuous with respect to u. If the controlsystem is locally controllable at x0 then for every T , ε 0 one has:Rx0 (T ) Int(Rx0 (T ε)).(7)Proof. Consider x Rx0 (T ) and let ux : [0, T ] U be such that the correspondingtrajectory starting from x0 reaches x at time T . Moreover, let Φt be the flux associatedto the time varying vector field f (·, u(t)) and notice that Φt is a diffeomorphism. Bylocal controllability at x0 , Rx0 (ε) is a neighborhood of x0 . Thus ΦT (Rx0 (ε)) is aneighborhood of x and, using ΦT (Rx0 (ε)) Rx0 (T ε), we conclude. 3Optimal ControlIn this section we give an introduction to the theory of Optimal Control. Optimal Control can be seen as a generalization of the Classical Calculus of VariationsRT(min 0 L(x(t), ẋ(t))) to systems with nonholonomic constrains of the kind ẋ f (x, u), u U .Remark 4 We recall that a constraint on the velocity is said to be nonholonomic if itcannot be obtained as consequence of a (holonomic) constraint on the position of thekind:ψi (x) 0, i 1, ., n0 , n0 n.where the real functions ψi (x) are sufficiently regular to define a submanifold M 0 M . Clearly since holonomic constraints can be eliminated simply by restricting theproblem to M 0 , the interesting case is the nonholonomic one.In the followingwhen we speak about nonholonomic constraints we always refer to nonholonomicconstraints of the kind ẋ f (x, u), u U .The most important and powerful tool to look for an explicit solution to an OptimalControl Problem is the well known Pontryagin Maximum Principle (in the followingPMP, see for instance [3, 29, 49]) that give a first order necessary condition for optimality. PMP is very powerful for the following reasons: it generalizes the Euler Lagrange equations and the Weierstraß condition of Calculus of Variations to variational problems with nonholonomic constrains; it provides a pseudo-Hamiltonian formulation of the variational problem in thecase in which the standard Legendre transformation is not well defined (as inthe case of Optimal Control, see below).Roughly speaking PMP says the following. If a trajectory of a control system is aminimizer, then it has a lift to the cotangent bundle, formed by vector-covector pairs,such that:
26U. B OSCAINANDB. P ICCOLI it is a solution of an pseudo-Hamiltonian system, the pseudo Hamiltonian satisfies a suitable maximization condition.Here we speak of a pseudo-Hamiltonian system since the Hamiltonian depends on thecontrol (see below). In the regular cases the control is computed as function of thestate and costate using the maximization condition.It is worth to mention that giving a complete solution to an optimization problem (that for us means to give an optimal synthesis, see Definition 7) in general isextremely difficult for several reasons:A the maximization condition not always provide a unique control. Moreover PMPgives a two point boundary value problem with some boundary conditions givenat initial time (state) and some given at final time (state and covector);B one is faced with the problem of integrating a pseudo–Hamiltonian system (thatgenerically is not integrable except for very special dynamics and costs);C a key role is played by some special classes of extremals called abnormal (extremals independent from the cost) and singular (extremals that are singularityof the End-Point Mapping). See Section 3.2.3;D even if one is able to find all the solutions of the PMP it remains the problem ofselecting among them the optimal trajectories.Usually A, B and C are very complicated problems and D may be even more difficult.For this reason (out from the so called linear quadratic problem, see for instance[3, 29]) one can hope to find a complete solution of an Optimal Control Problem onlyin low dimensions, unless the system presents a lot of symmetries. For instance mostof the problems in dimension 3 are still open also for initial and final conditions closeone to the other.In Section 3.1 we give a brief introduction to the PMP in Optimal Control as ageneralization of the classical first order conditions in Calculus of Variations.In Section 3.2 we briefly discuss the problem of existence of minimizers, statethe PMP, define abnormal and singular trajectories. In Section 3.3 we show how toget, in the case of the Calculus of Variations, the Euler Lagrange equations and theWeierstraß condition. In Section 3.4, as an application we show in some detail howto compute geodesics for a famous singular Riemannian problem using the PMP. InSection 3.5 the problem of constructing an Optimal Synthesis with geometric methodsis treated.3.1IntroductionIn this Section we first recall the Euler Lagrange equations, how to transform them inan Hamiltonian form and discuss in which case this transformation is applicable (theLegendre transformation must be invertible).
A N I NTRODUCTIONTOO PTIMAL C ONTROL27In Section 3.1.2 it is shown that for nonholonomic constraints (i.e. for an Optimal Control problem with nontrivial dynamics) the Euler Lagrange equation cannotbe written in standard form and the Legendre transformation is never well defined.Finally we explicate some connection with the Lagrangian formulation of mechanicalsystems.Here, for simplicity, the state space is assumed to be Rn and not an arbitrarymanifold.3.1.1 The Legendre TransformationConsider a standard problem in Calculus of Variations: RT minimizeL(x(t), ẋ(t))dt,0x(0) x0 , x(T ) xT ,(8)where x (x1 , ., xn ) Rn and L : R2n R is a C 2 function. It is a standard factthat if a C 2 minimizer x(.) exists, then it must satisfy the Euler Lagrange equations:d L L ,dt ẋi xithat for our purpose is better to write more precisely as:! L(x, u)d L(x, u) .dt ui xi(x(t),ẋ(t))(x(t),ẋ(t))(9)(10)(11)Euler Lagrange equations are second order ODEs very difficult to solve in general,also numerically. In fact, the theory of differential equation is much more developedfor first order than for second order differential equations. For this reason it is often convenient to transform equations (10) into a system of ODEs of a special form(Hamiltonian equations) via the so called Legendre transformation.The problem of finding solutions to a system of ODEs is simplified if the systemadmits constants of the motion (in involution). Roughly speaking this permits to decouple the problem to the corresponding level sets. The most important advantage ofpassing from the Lagrangian to the Hamiltonian formulation is that, in Hamiltonianform, it is easier to recognize constants of the motion.The Legendre transformation consists in the following. We first reduce the system (10) of n second order ODEs to a system of 2n firstorder ODEs introducing the variable u : ẋ: d L(x,u) L(x,u)dt ui xi(12)(x(t),u(t))(x(t),u(t)) ẋ(t) u(t).
28U. B OSCAINANDB. P ICCOLI Then we make the change of coordinates in Rn :(x, u) (x, p), where pi Φi (x, u) : L(x, u). uiThis change of coordinates is well defined if it realizes a C 1 -diffeomorphism ofR2n into R2n , i.e. we must have: 2011 L(x, u)det det6 0,(13) Φ(x,u) Φ(x,u) ui uj. x ufor every (x, u) R2n . If the condition (13) is satisfied, the Legendre transformation is said invertible. In this case the inverse transformation is u Φ 1 (x, p) and the Lagrangian is called regular. Define the function (called Hamiltonian):(14)H(x, p) : pΦ 1 (x, p) L(x, Φ 1 (x, p)).In the (x, u) coordinates the Hamiltonian takes the form L u u L(x, u) andusually one remembers it in the “mixed coordinates” form pu L.After the Legendre transformation (if it is invertible), the Euler Lagrange equations are written as the Hamiltonian equations:( Hẋ pṗ H x(15)In fact using carefully the chain rule we have for the first: pΦ 1 (x, p) L(x, Φ 1 (x, p)) p L(x, u) Φ 1 (x, p) Φ 1 (x, p) p p uẋ and using the fact that u Φ 1 (x, p) and p Similarly for the second of (15) we have: L u(x,Φ 1 (x,p)) Φ 1 (x, p), pwe get the second of (
AN INTRODUCTION TO OPTIMAL CONTROL 23 Definition 5 (Lie Algebra of F) Let F be a family of smooth vector fields on a smooth manifold Mand denote by (M)the set of all C1 vector fields on M. The Lie algebra Lie(F) generated by F is the smallest Lie subalgebra of (M) containing
II. Optimal Control of Constrained-input Systems A. Constrained optimal control and policy iteration In this section, the optimal control problem for affine-in-the-input nonlinear systems with input constraints is formulated and an offline PI algorithm is given for solving the related optimal control problem.
not satisy \Dynamic Programming Principle" (DPP) or \Bellman Optimality Principle". Namely, a sequence of optimization problems with the corresponding optimal controls is called time-consistent, if the optimal strategies obtained when solving the optimal control problem at time sstays optimal when the o
optimal control, dynamic programming, Pontryagin maximum principle. I. INTRODUCTION he optimal control of HEVs (Hybrid Electric Vehicles) is an important topic not only because it is useful for power-management control but also indispensible for the optimal des
work/products (Beading, Candles, Carving, Food Products, Soap, Weaving, etc.) ⃝I understand that if my work contains Indigenous visual representation that it is a reflection of the Indigenous culture of my native region. ⃝To the best of my knowledge, my work/products fall within Craft Council standards and expectations with respect to
a MDP in the domain of transportation optimal control. To solve a DEDP analytically, we derived a duality theorem that recasts optimal control to variational inference and param-eter learning, which is an extension of the current equivalence results between optimal control and probabilistic inference [24, 38] in Markov decision process research.
The optimal control problem of the isolated subsystems is described under the framework of HJB equations. The decentra-lized control law is derived by adding some local feedback gains to the isolated optimal control policies. 3.1. Optimal control In this paper, to design the decentralized control law, we need
4. Linear-quadratic-Gaussian control, Riccati equations, iterative linear approximations to nonlinear problems. 5. Optimal recursive estimation, Kalman –lter, Zakai equation. 6. Duality of optimal control and optimal esti
compared to the three previous methods. ’ Some previous algorithms achieve optimal mapping for restricted problem domains: Chortle is optimal when the input network is a tree, Chortle-crf and Chortle-d are optimal when the input network is a tree and h’ 5 6, and DAG- Map is optimal when the mapping constraint is monotone, which is true for .
