Convergence rates of moment-sum-of-squares hierarchies for optimal control problems

Transcription

1 Convergence rates of moment-sum-of-squares hierarchies for optimal control problems Milan Kora 1, Diier Henrion 2,3,4, Colin N. Jones 1 Draft of September 8, 2016 Abstract We stuy the convergence rate of moment-sum-of-squares hierarchies of semiefinite programs for optimal control problems with polynomial ata. It is known that these hierarchies generate polynomial uner-approximations to the value function of the optimal control problem an that these uner-approximations converge in the L 1 norm to the value function as their egree tens to infinity. We show that the rate of this convergence is O1/ log log ). We treat in etail the continuous-time infinite-horizon iscounte problem an escribe in brief how the same rate can be obtaine for the finite-horizon continuous-time problem an for the iscrete-time counterparts of both problems. Keywors: optimal control, moment relaxations, polynomial sums of squares, semiefinite programming, approximation theory. 1 Introuction The moment-sum-of-squares hierarchy also know as Lasserre hierarchy) of semiefinite programs was originally introuce in [10] in the context of polynomial optimization. It allows one to solve globally non-convex optimization problems at the price of solving a sequence, or hierarchy, of convex semiefinite programming problems, with convergence guarantees; see e.g. [13] for an introuctory survey, [11] for a comprehensive overview an [3] for control applications. This hierarchy was extene in [12] to polynomial optimal control, an later on in [6] to global approximations of semi-algebraic sets, originally motivate by volume an integral estimation problems. The approximation hierarchy for semi-algebraic sets erive in [6] was 1 Laboratoire Automatique, École Polytechnique Féérale e Lausanne, Station 9, CH-1015, Lausanne, Switzerlan. {milan.kora,colin.jones}@epfl.ch 2 CNRS; LAAS; 7 avenue u colonel Roche, F Toulouse; France. henrion@laas.fr 3 Université e Toulouse; LAAS; F Toulouse; France. 4 Faculty of Electrical Engineering, Czech Technical University in Prague, Technická 2, CZ Prague, Czech Republic. 1

2 then transpose an aapte to an approximation hierarchy for transcenental sets relevant for systems control [2], such as regions of attraction [7] an maximal invariant sets for controlle polynomial ifferential an ifference equations [9], still with rigourous analytic convergence guarantees. Central to the moment-sum-of-squares hierarchies of [12, 7, 9] are polynomial subsolutions of the Hamilton-Jacobi-Bellman equation, proviing certifie lower bouns, or unerapproximations, of the value function of the optimal control problem. It was first shown in [12] that the hierarchy of polynomial subsolutions of increasing egree converges locally i.e. pointwise) to the value function on its omain. Later on, as an outcome of the results of [7], global convergence i.e. in L 1 norm on compact omains, or equivalently, almost uniformly) was establishe in [8]. The current paper is motivate by the analysis of the rate of convergence of the momentsum-of-squares hierarchy for static polynomial optimization achieve in [14]. We show that a similar analysis can be carrie out in the ynamic case, i.e. for assessing the rate of convergence of the moment-sum-of-squares hierarchy for polynomial optimal control. For ease of exposition, we focus on the iscounte infinite-horizon continuous-time optimal control problem an briefly escribe in Section 5) how the same convergence rate can be obtaine for the finite-time continuous version of the problem an for the iscrete counterparts of both problems. Our main Theorem 4 gives estimates on the rate of convergence of the polynomial unerapproximations to the value function in the L 1 norm. As a irect outcome of this result, we erive in Corollary 2 that the rate of convergence is in O1/ log log ), where is the egree of the polynomial approximation. As far as we know, this is the first estimate of this kin in the context of moment-sum-of-squares hierarchies for polynomial optimal control. 1.1 Notation The set of all continuous functions on a set X R n is enote by CX); the set of all k- times continuously ifferentiable functions is enote by C k X). For h CX), we enote h C 0 X) := max x X hx) an for h C 1 X) we enote h C 1 X) := max x X hx) + max x X hx) 2 where h is the graient of h. The L 1 norm with respect to a measure µ 0 of a measurable function h : R n R is enote by h L1 µ 0 ) := hx)µ R n 0 x). The set of all multivariate polynomials in a variable x of total egree no more than is enote by R[x]. The symbol R[x] n enotes the n-fol cartesian prouct of this set, i.e., the set of all vectors with n entries, where each entry is a polynomial from R[x]. The interior of a set X R n is enote by IntX). 2

3 2 Problem setup Consier the iscounte infinite-horizon optimal control problem V x 0 ) := inf u ), x ) s.t. 0 e βt lxt), ut)) t xt) = x 0 + t fxs), us)) s t [0, ) 0 xt) X, ut) U t [0, ) 1) where β > 0 is a given iscount factor, f R[x, u] n f an l R[x, u] l are given multivariate polynomials an the state an input constraint sets X an U are of the form X = {x R n g X i x) 0, i = 1,..., n X }, U = {u R m g U i u) 0, i = 1,..., n U }, where g X i R[x] X i an g U i R[u] U i are multivariate polynomials. The function V in 1) is calle the value function of the optimal control problem 1). Let us recall the Hamilton-Jacobi-Bellman inequality lx, u) βv x, u) + V x, u) fx, u) 0 x, u) X U 2) which plays a crucial role in the erivation of the convergence rates. In particular, for any function V C 1 X) that satisfies 2) it hols V x) V x) x X. 3) The following polynomial sum-of-squares optimization problem provies a sequence of lower bouns to the value function inexe by the egree : max V x) µ V R[x] X 0x) 4) s.t. l βv + V f Q +f X U), where µ 0 is a given probability measure supporte on X e.g., the uniform istribution), an Q +f X U) := { s 0 + n X i=1 g X i s i X + n U i=1 g U i s i U : s 0 Σ +f )/2, s i X Σ +f i X )/2, s i U Σ +f i U )/2 }, is the truncate quaratic moule associate with the sets X an U see [13] or [11]), where Σ is the cone of sums of squares of polynomials of egree up to. Note that whenever V is feasible in 4), then V satisfies Bellman s inequality 2), because polynomials in Q +f X U) are non-negative on X U by construction. Therefore any polynomial V feasible in 4) satisfies also 3) an hence is an uner-approximation of V on X. The truncate quaratic moule is essential to the proof of convergence of the momentsum-of-squares hieararchy in the static polynomial optimization case [10] which is base on Putinar s Positivstellensatz [15]. We recall that some polynomials of egree + f nonnegative on X U may not belong to Q +f X U) [11]. On the other han, optimizing 3

4 over the polynomials belonging to Q +f X U) is simple it translates to semiefinite programming) while optimizing over the cone of non-negative polynomials is very ifficult in general. In particular, the optimization problem 4) translates to a finite-imensional semiefinite programming problem SDP). The fact that the truncate quaratic moule has an explicit SDP representation an hence can be tractably optimize over is one of the main reasons for the popularity of the moment-sum-of-squares hierarchies across many fiels of science. Throughout the paper we impose the following staning assumptions. Assumption 1 The following conitions hol: a) X [ 1, 1] n an U [ 1, 1] m. b) The sets of polynomials g X i ) n X i=1 an g U i ) n U i=1 both satisfy the Archimeian conition1. c) 0 IntX) an 0 IntU). ) The function V is Lipschitz continuous on X. e) The set fx, U) is convex for all x X an the function v inf {lx, u) v = fx, u)} u U is convex for all x X. The Assumption a) an b) are mae without loss of generality since the sets X an U are assume to be compact an hence can be scale such that they are inclue in the unit ball; aing reunant ball constraints 1 x 2 an 1 u 2 in the escription of X an U then implies the Archimeianity conition. Assumption c) essentially requires that the sets X an U have nonempty interiors a mil assumption) since then a change of coorinates can always be carrie out such that the origin is in the interior of these sets. Assumption ) is an important regularity assumption necessary for the subsequent evelopments. Assumption e) is a stanar assumption ensuring that the value function of the so-calle relaxe formulation of the problem 4) coincies with V see, e.g., [18]) an is satisfie, e.g., for input-affine 2 systems with input-affine cost function provie that U is convex. This class of problems is by far the largest an practically most relevant for which this assumption hols although other problems exist that satisfy this assumption as well 3. Uner Assumption 1, the hierarchy of lower bouns generate by problem 4) converges from below in the L 1 norm to the value function V ; see e.g. [8]: Theorem 1 There exists a 0 0 such that the problem 4) is feasible for all 0. In aition V V for any V feasible in 4) an lim V V L 1 µ 0 ) = 0, where V is an optimal solution to 4). The goal of this paper is to erive bouns on the convergence rate of V to V. 1 A sufficient conition for a set of polynomials g i ) n i=1 to satisfy the Archimeian conition is g i = N x 2 2 for some i an some N 0, which is a non-restrictive conition provovie that the set efine by g i s is compact an an estimate of its imeter is known. For a precise efinition of this conition see Section of [13]. 2 A system is input-affine if fx, u) = f x x) + f u x)u for some functions f x an f u. 3 For example, consier lx, u) = x 2, fx, u) = x + u 2, U = [ 1, 1]. 4

5 3 Convergence rate The convergence rate is a consequence of the following funamental results from approximation theory an polynomial optimization. Theorem 2 Bagby et al. [1]) If h : X R is a function such that h C 1 X), then there exists a sequence of polynomials p ) =1 satisfying egp ) such that h p C 1 X) c 1 / for some constant c 1 0 epening on h an X only. Now we turn to the secon funamental result. Given a polynomial p R[x] expresse in a multivariate monomial basis as px) = α N n α with α = n i=1 α i an x α = n i=1 xα i, we efine p R[x] = max α β α x α β α ), 5) α α where the multinomial coefficient ) α α is efine by ) α α! := α α 1!... α n!. Theorem 3 Nie & Schweighofer [14]) Let p R[x, u] p an let p min := Then p Q X U) provie that min px, u) with p min > 0. x,u) X U c 2 exp 2 pn + m) p ) p R[x,u] c2, 6) p min where the constant c 2 epens only on the sets X an U. In the following evelopments it will be crucial to boun the norm R[x] of a polynomial by its supremum norm CX). We remark that such a boun is possible only for a generic set X such that any polynomial vanishing on X necessarily vanishes everywhere. A sufficient conition for this is IntX). This is the reason for Assumption 1 c). Lemma 1 If p R[x], x R n, then p R[x] 3 +1 p C[ 1,1] n ) 7) for all 0. 5

6 Proof: The iea is to use a multivariate Markov inequality to boun the erivatives of the polynomial at zero an hence its coefficients) in terms of its supremum norm on [ 1, 1] n. Let p = α β αx α R[x]. From [17, Theorem 6], we have α p x 0) α ) T α 0) + is α ) 0) p C[ 1,1] n ) for all multiinices α satisfying α, where i = 1, T y) = cos arccosy)), y [ 1, 1], enotes the -th univariate Chebyshev polynomial of the first kin, S y) = sin arccosy)) = 1 1 x 2 T y), y [ 1, 1], an hk) signifies the k-th erivative of a function h : R R. It is easy to see that S k) 0) = 1 T k+1) 0) an hence T α ) 0) + is α ) 0) T α ) 0) + 1 [ α! + α + 1)! T α +1) 0) = α! t, α + ] t, α + 1)! t, α +1 where t,k enotes the k-th coefficient of T when expresse in the monomial basis i.e., T y) = k=0 t,ky k ) an t = max k {0,...,} t,k. Since β α = α 1!... α n!) 1 α p x α 0), we get β α ) = α 1!... α n!) β α = 1 α p [ α! α! x 0) 1 + α + 1 ] t α p C[ 1,1] n ). α α In view of 5) an since α we get p R[x] [ ] t p C[ 1,1] n ). It remains to boun t. From the generating recurrence of T +1 y) = 2yT y) T 1 y) starting from T 0 = 1 an T 1 = y, it follows that t t, where t solves the linear ifference equation t +1 = 2 t + t with the initial conition t 0 = 1 an t 1 = 1. The solution to this equation is t = 1 + 2) ) ) 2) 2 3, 1. 2 Therefore t 3 for 1 an hence [ p R[x] ] 3 p C[ 1,1] n ) 3 +1 p C[ 1,1] n ), 1. Since p R[x] = p C[ 1,1] n ) for = 0, the result follows. In orer to state an immeiate corollary of this result, crucial for subsequent evelopments, we efine 1 r := sup{s > 0 [ s, s] n+m X U}, 8) which is the reciprocal value of the length of the sie of the largest box centere aroun the origin inclue in X U. By Assumption 1 a) an c), we have r [1, ). 6

7 Corollary 1 If p R[x, u], then where k) = 3 +1 r with r efine in 8). Proof: Set px, u)) := pr 1 x, u)). Then we have p R[x,u] k) p CX U), 9) p C[ 1,1] n+m ) = p C[ 1/r,1/r] n+m ) p CX U) 10) since [ 1/r, 1/r] n+m X U by efinition of r 8). In aition p R[x,u] = max β α α r α α ) = r max β α α r α α ) r p R[x,u]. 11) Combining 10), 11) an Lemma 1 we get α p R[x,u] r p R[x,u] 3 +1 r p C[ 1,1] n+m ) 3 +1 r p CX U) = k) p CX U). as esire. Now we turn to analyzing the Bellman inequality 2). The following immeiate property of this inequality will be of importance: α Lemma 2 Let V satisfy 2) an let a R. Then Ṽ := V a satisfies l βṽ + Ṽ f βa x, u) X U. Proof: We have l βṽ + Ṽ f = l βv + V f + βa βa, since V satisfies 2). We will also nee a result which estimates the istance between the best polynomial approximation of a given egree to the value function an polynomials of the same egree satisfying Bellman s inequality. A similar result in iscrete time an with iscrete state an control spaces can be foun in [5]. Lemma 3 Let ˆV arg min V V C 1 X). V R[x] Then there exists a polynomial Ṽ R[x] satisfying 2) an such that Ṽ V C 1 X) ˆV V C 1 X) 2 + f C 0 X) β ). 12) Proof: Let Ṽ := ˆV a. We will fin an a 0 such that Ṽ satisfies the Bellman inequality. We have l βṽ + Ṽf = l β ˆV + ˆV f + βa = l βv + V f + βv ˆV ) + ˆV V )f + βa βv ˆV ) + ˆV V )f + βa β ˆV V C 1 X) ˆV V C 1 X) f C 0 X) + βa, 7

8 an hence if a := 1 + f C 0 X) β ) ˆV V C 1 X), then Ṽ satisfies Bellman s inequality 2) an estimate 12) hols. Now we are in position to prove our main result which bouns the gap, in L 1 norm, between the value function V of the optimal control problem 1) an any optimal solution V of the sum-of-squares program 4): Theorem 4 It hols that V V L 1 µ 0 ) < ɛ for all integer where p = 2c 1 [ c 2 exp = O exp [ 1 ɛ 2+ f C 0 β 6 p ɛ) 2 3rn + m)) pɛ) M + βɛ + δ 1 f C 0 βɛ ) c2 ] 13) ɛ 3n + m)r) ] ) c3 ɛ, 14) 3c 2 )+ f, M = l βv + V f C 0 X U) <, r is efine in 8), c 3 = 2c 1 c 2 2β + f C 0)/β, the constant c 1 epens only on V an X an U, whereas the constant c 2 epens only on sets X an U. Proof: Accoring to Theorem 2 an Lemma 3 we can fin a polynomial Ṽ of egree no more than 2c1 = 2 + f ) C 0 ɛ β such that V Ṽ C 1 ɛ an such that Ṽ 2 satisfies the Bellman inequality 2). Let V be an arbitrary polynomial feasible in 4) for some 0. Then V V L1 V V L1 V V C 0 V Ṽ C 0+ V Ṽ C 0 ɛ 2 + V Ṽ C 0. 15) Hence, the goal is to fin a egree 0 an a polynomial V feasible in 4) for that satisfying V Ṽ C 0 ɛ/2. Setting V := Ṽ ɛ/2, we clearly have V Ṽ C 0 ɛ/2; in aition, using Lemma 2 we know that l βv + V f 1 βɛ > 0 16) 2 an hence V strictly satisfies the Bellman inequality an as a consequence of the Putinar s Positivstellensatz [15] there exists a egree 0 such that V is feasible in 4). To boun the egree we apply the boun of Theorem 3 on p := l βv + V f. From 16) we know that p min 1 2 βɛ. Next, we nee to boun p R[x,u] by bouning p C 0 X U) an using Corollary 1. We have p C 0 = l βv + V f C 0 = l βṽ + Ṽ f βɛ C 0 l βv + V f + C 0 + β V Ṽ C 0 + V Ṽ C 1 f C βɛ M βɛ ɛ f C βɛ = M + βɛ ɛ f C 0. 8

9 Finally, we nee to estimate the egree of p. We have assuming without loss of generality that + f 1 egl)) egp) = eg l βv + V f ) 2c1 + f f ) C 0 + f ɛ β ) 2c Setting p := 1 ɛ 2 + f C 0 + β f an using Theorem 3 an Corollary 1, we conclue that for [ c 2 exp 2 2 pn + m) p k p)m + βɛ + δ 1 f C 0) βɛ ) c2 ], the polynomial V is feasible in 4). Since Ṽ V C 0 ɛ, we conclue from 15) that 2 V V L 1 ɛ. Inserting the expression for k) = 3 +1 r from Corollary 1 yiels 13) an carrying out asymptotic analysis for ɛ 0 yiels which is 14). O exp [ 1 ɛ 3c 2 3n + m)r) c3 ɛ ] ), Corollary 2 It hols V V L 1 µ 0 ) = O1/ log log ). Proof: Follows by inverting the asymptotic expression 14) using the fact that for small ɛ. 3n + m)r) 2c 3 ɛ 1 ɛ 3c 2 3n + m)r) c3 ɛ 4 Discussion The boun on the convergence rate O1/ log log ) shoul be compare with the boun O1/ c 2 log ) erive in [14] for static polynomial optimization problems here c2 1 is the, in general, unknown constant from Theorem 3). The aitional logarithm appearing in our boun seems to be unavoiable ue to funamental results of approximation theory known as Bernstein inequalities) implying that Lipschitz continuous functions cannot be approximate by polynomials with rate faster than 1/ in the sense that there exists a Lipschitz continuous function whose best egree- approximation converges to f with the rate exactly C/, C > 0, in the supremum norm on [ 1, 1] n ); this implies the 1/ɛ epenence of p from Theorem 4 which then propagates to oubly exponential epenence on 1/ɛ through Theorem 3. Therefore the primary point of improvement of the boun from Theorem 4 an Corollary 2 is the funamental boun of Theorem 3 erive in [14]. As the authors of [14] remark, this boun is far from tight, at least in two special cases: the univariate case i.e., n+m = 1 in our setting) or the case of a single constraint efining X U. In these cases the exponential in 6) can be roppe, which results in O1/ log ) asymptotic rate of convergence in Corollary 2. In the general case, however, it is unknown whether the exponential in 6) can be roppe or whether the boun 6) can be improve otherwise [14]. 9

10 5 Extensions The approach for eriving this boun can be extene to other settings. In particular, similar bouns, with ientical O1/ log log ) asymptotics, hol for the finite-horizon version of the problem, both in continuous an iscrete time, as well as for the iscounte iscrete-time infinite-horizon variant the former was treate using the moment-sum-of-squares approach, in continuous time, in [12] an the latter was treate in [16]). The erivation in iscrete-time is completely analogous an the results hol uner miler assumptions Assumption 1 ) can be replace by V Lipschitz an Assumption 1 e) can be roppe completely). For the finite-horizon continuous-time problem, the only ifference is in Lemma 3, where the constant shift Ṽ x) = ˆV x) a, is replace by the affine shift Ṽ t, x) = ˆV t, x) a bt t) for suitable a > 0, b > 0 ensuring that Ṽ satisfies the corresponing finite-time Bellman inequality an its bounary conition hence the two egrees of freeom). References [1] T. Bagby, L. Bos, N. Levenberg. Multivariate simultaneous approximation. Constructive approximation, 18: , [2] G. Chesi. Domain of attraction; analysis an control via SOS programming. Lecture Notes in Control an Information Sciences, Vol. 415, Springer-Verlag, Berlin, [3] G. Chesi. LMI techniques for optimization over polynomials in control: a survey. IEEE Transactions on Automatic Control, 55: , [4] E. e Klerk, R. Hess, M. Laurent. Improve convergence rates for Lasserretype hierarchies of upper bouns for box-constraine polynomial optimization. arxiv: , March [5] D. P. e Farias, B. Van Roy. The Linear Programming Approach to Approximate Dynamic Programming. Operations Research, 51: [6] D. Henrion, J. B. Lasserre, C. Savorgnan. Approximate volume an integration for basic semialgebraic sets. SIAM Review 514): , [7] D. Henrion, M. Kora. Convex computation of the region of attraction of polynomial control systems. IEEE Transactions on Automatic Control, 59: , [8] D. Henrion, E. Pauwels. Linear conic optimization for nonlinear optimal control. arxiv: , July [9] M. Kora, D. Henrion, C. N. Jones. Convex computation of the maximum controlle invariant set for polynomial control systems. SIAM Journal on Control an Optimization, Vol. 52, No. 5, pp , [10] J. B. Lasserre. Optimisation globale et théorie es moments. C. R. Aca. Sci. Paris. Série I, 331:929934,

11 [11] J. B. Lasserre. Moments, positive polynomials an their applications. Imperial College Press, Lonon, UK, [12] J. B. Lasserre, D. Henrion, C. Prieur, E. Trélat. Nonlinear optimal control via occupation measures an LMI relaxations. SIAM Journal on Control an Optimization, 47: , [13] M. Laurent. Sums of squares, moment matrices an polynomial optimization. In M. Putinar, S. Sullivan es.). Emerging applications of algebraic geometry, Vol. 149 of IMA Volumes in Mathematics an its Applications, Springer, Berlin, [14] J. Nie, M. Schweighofer. On the complexity of Putinar s Positivstellensatz. Journal of Complexity 23: , [15] M. Putinar. Positive polynomials on compact semi-algebraic sets. Iniana University Mathematics Journal, 42: , [16] C. Savorgnan, J. B. Lasserre, M. Diehl. Discrete-time stochastic optimal control via occupation measures an moment relaxations. Proceeings of the 48th IEEE Conference on Decision an Control, [17] V. I. Skalyga. Analogs of the Markov an Schaeffer-Duffin inequalities for convex boies. Mathematical Notes 68: , [18] R. Vinter. Convex uality an nonlinear optimal control. SIAM Journal on Control an Optimization, 31:518:538,