Supplementary material for Continuous-action planning for discounted infinite-horizon nonlinear optimal control with Lipschitz values

Transcription

1 Supplementary material or Continuous-action planning or discounted ininite-horizon nonlinear optimal control with Lipschitz values List o main notations x, X, u, U state, state space, action, action space, u dynamics, sequence o actions γ, r, ρ, v discount actor, reward, reward cn., value L,L ρ,l v Lipschitz constants o respective cns. U box in the space o action sequences i,k;i,k box and dimension indices; selected indices K number o discretized box dimensions µ, d action interval, interval length n computation budget M number o subintervals or splitting T, T, L tree, near-optimal tree, set o leaves h,s depth in tree, number o splits bi), δi) b-value, diameter o box i m branching actor o near-optimal tree h max n) maximum depth unction, c, C constants O ),Õ ) bounded by up to const., log. terms ceiling smallest integer still larger than ) diag[ ] diagonal matrix with on diagonal A Proos Proo o Lemma 3: We will prove the inequality rom the theorem or any two sequences u and u, which directly implies it or the particular case u = u rom 3). Denote by x k, x k, r k, r k or k the state and reward trajectories generated by the two sequences, with x = x. Then, by induction: k 1 x k x k Indeed, at k = 1, we have: j= L k j uj u j A.1) x 1 x 1 L x x + u u ) = L u u and assuming the relation holds at k 1, we get: x k x k L xk 1 x k 1 + uk 1 u k 1 ) k L j= L k 1 j equivalent to A.1). Then: uj u j + L uk 1 u k 1 vu ) vu ) γ k r k r k γ k L ρ x k x k + u k u k ) L ρ γ k k j= L k j u j u j = L ρ u k u k γ k+j L j L ρ = 1 γl j= γ k u k u k Here, we used the Lipschitz continuity o ρ, and then applied A.1). The three equalities simply rewrite the right hand side, isolating the contribution o each u k u k ; the middle equality swaps indices k and j, and can be veriied by writing out explicitly the summations. The last step holds because γl < 1. Thus the inequality is proven with L v = Lρ 1 γl. Proo o Theorem 6: Our goal is to ind an upper bound or the diameter o an arbitrary box i at some depth h. Since we deal with this single box, we will omit its index i or most o the derivation. Indeed, due to dimension selection 7), all the boxes at a given depth have the same shape. Recall unction sk), the number o splits per dimension k, which is clearly decreasing, see Fig. A.1 or an example. In addition s decreases in steps o at most 1. To see this, consider k like in the igure, the irst dimension in a constant-s range, which will always be preerred to later dimensions in the same range. To increase the gap to, dimension k 1 must be expanded beore k, which means γ k 1 M sk)+1) γ k M sk), or M 1/γ. This contradicts Assumption 5ii). Denote now the lengths o the ranges in s by, 1,..., N where N is the last, ininitely long range where s =. Let j be the index o the range starting with k. By direct computation like above, we ind that i node k is expanded: j log M log 1/γ, j 1 < log M log 1/γ i j 1 A.) Recall that = log M log 1/γ, so A.) implies j, j 1 < 1. Keeping in mind that s and so N, j ) depend on h, we prove by induction that at any depth h, we have: j { 1,} N = or 1 j < N A.3)

2 For any, the irst h = 3 + are done in the same order, shown in the igure. To see this, consider any step and try to expand any other dimension; one inds that A.) is violated. Clearly any unction s so ar obeys A.3). Consider now an arbitrary h 3+ that satisies A.3). To study h + 1, our cases must be considered or the next expanded dimension, denoted a)-d) inside dotted squares in the igure. Call the next unction s, with N ranges j. Case a): Since k = 1 is expanded, = due to A.), A.3). Hence = 1, 1 = 1, and later ranges remain unchanged. Case b): We must have 1, 1 =. Hence,, 1 = 1, and the later ranges are unchanged. Case c): Some arbitrary k in some range 1 < j < N is expanded. We must have j 1 =, j = 1, leading to j 1 = 1, j =, and the other ranges remain the same. Case d): Finally, when the irst undiscretized dimension K is split, we have N 1 = 1, N =, leading to N = N, N 1 =, N =, with no change to the prior ranges. In all cases, the new unction satisies A.3), so the induction is complete. The next step is to ind, or ixed h, a lower bound on sk) under constraints 4), A.3). I the length K o the box is number o splits s k) a 3+1 b j-1 { k c j { d K dimension Fig. A.1. An example o a split unction is shown as a thick black line. The irst unsplit dimension is K, and a dimension k is highlighted with the constant-s ranges to its let and right, j 1 and j. Each square at a certain value o k indicates a split o dimension k, and the squares stack together to obtain sk). The irst 3 + splits, shown on the bottom-let, always occur in the same order, indicated by indices inside the squares. The dotted squares above sk) indicate possibilities or the next split o sk). number o splits s 1 s K... s K -1-1 K K K dimension Fig. A.. Various split unction bounds used in the proo, see there or their meaning. Dotted: s K or arbitrary K; gray: s yielding K; dashed: s K where K K. ixed and 4) is ignored thereby relaxing the problem), then a lower bound s K on any unction or this K is obtained by illing in s with ranges o ; see Fig. A.. Now s K k) is decreasing with K or any k, so we must ind a lower bound on K, denoted K. To this end, we ill in a unction s with all ranges o 1, except which is equal to 1, see again Fig. A.; while imposing a relaxed version o 4), namely h sk). Denote N the number o ranges in s, then by explicit computation we get: h sk) = N 1)N 1)+N rom which we get N computation and replacing N: h 1 N + 1) 1) 1 =: N. Again by K = + N 1) 1) 4 + h 1) =: K K k Since s K = K k or k < K, we inally obtain a lower-bound split unction as s K, which at k < K is explicitly: s K = and or k K. h 1 k + 4/ ) To complete the proo, we use this unction in the diameter ormula: δi) = L v γ k M sk) K 1 L v γ k M s K + γk 1 γ [ L ] h v c 1 h 1)M 1 + c γ h 1 c h 1)γ h 1 where we have omitted some tedious derivations. An intermediate step was highlighted to show that some conservativeness is introduced in the tail term, notably by decreasing its rate o convergence via the division by. Here, c 1,c,c denote positive constants whose value is not important in the asymptotic analysis. Proo o Lemma 8: For the irst part, consider any iteration t, and denote by i t the box expanded at this iteration. Since the lea boxes on the current tree T cover the entire space, there exists a lea box j containing an optimal solution, or which bj) v by 6). As the expanded box i t is selected by maximizing the b-value,

3 bi t ) bj) v. Further, vi t ) + δ h bi t ), where h is the depth o i t, and thereore inally i t T. For the second part, among the descendants o i t there exists a lea j on the inal tree so that vj) vi t ), due to Assumption 5i). Since û maximizes the value among the leaves, vû) vj) vi t ). Combining this with bi t ) v rom the irst part, we get v vû)) bi t ) vi t ) = δi t ). Since this holds at any iteration, the bound δ min = min t δi t ) ollows. Proo o Theorem 9: For a given budget n the branching actor is used to iner a lower bound on the depth reached by OPC. For m > 1, take the smallest depth h so that Cm h+1 Mh n. The let-hand side is chosen since there are less than Cm h+1 nodes in the explored tree up to depth h, and each o them takes at most Mh computation to expand, see Fig. A.3. Thereore, we are sure h h T T h * up to up to Cm h Cm h+1 nodes nodes total M trajectories, h steps Fig. A.3. To reach depth h, at most Cm h nodes must be expanded on T h at depths h h, or a total o Cm h+1. Each o these expansions requires, at worst, simulating M trajectories with h steps. that at least a node at depth h min = h 1 has been expanded, and by solving the inequality we get ater some computation: h min log n log lognα)β log m log m or α,β > and suiciently large n. Replacing this in the ormula or δ hmin, we obtain: δ min δ hmin 1) log n 1 log m 1 γ )ˆ 1 1) log n log m 1 γ )ˆ 1 [ log n log lognα)β log m log m log lognα) β log m log lognα) β log m ) ] ) 1) log n γ log m [lognα)] β 1) log n log m where β > is a constant and we temporarily use notation zˆy = z y or readability. In the second step, the square root can be split in this way or suiciently large n; then we show that the second term enters in act the logarithmic, negligible part o the expression. When m = 1, take the smallest h so that Ch M n. Since at most C nodes must be expanded at each depth, this guarantees that some node was expanded at depth h min n/mc 1. Replacing this in the diameter ormula leads to the desired result. Note that we did not make explicit the term h min 1) in the diameter bounds. However, trivial upper bounds or h min are on the order o log n in the irst case since a tree with branching actor greater than 1 must asymptotically be explored, larger depths than this cannot be reached), and n in the second case. Both o these remain in the logarithmic part o the bound. Proo o Lemma 1: Consider one top-level loop o SOPC in Alg., and denote the number o elapsed such loops by l. The major part o the proo will be to show by induction that, or any h h max n), i l C h h =, mh then at least a box tree node) containing an optimal sequence at h has been expanded. For the base case, i l Cm 1, then the root has been expanded, and it contains the optimal solution by deinition. Take now an arbitrary h < h max n), and deine l h to be the loop where the irst node i h at h that contains an optimal solution was expanded. By the induction hypothesis, l h C h h =. Node i mh h has itsel some child i h+1 that contains an optimal solution. Note that another optimal node i h+1 may be expanded beore this particular child; we will be interested in i h+1 below.) Consider now any iteration l h +l where some other node i h+1, dierent rom i h+1, is expanded at depth h + 1. Then it must be that: vi h+1) vi h+1 ) v δi h+1 ) v δ h+1 where the middle inequality holds because i h+1 contains an optimal solution. But then i h+1 T h+1, and since there are at most Cm h+1 nodes in this set, including i h+1 which was not yet expanded, it means at most Cm h+1 1 loops can pass beore i h+1 must be expanded. Thus the loop l h+1 where i h+1 is expanded is, at worst, l h + Cm h+1 C h+1 h = mh, and the induction is proven. Observe next that each loop l expands at most h max n) nodes, where each node expansion takes at most

4 Mh max n) model calls. Then, by combining this with the bound on l obtained above, the algorithm is sure to expand an optimal node at hn) when this is smaller than h max n) or, when hn) > h max n), it expands an optimal node at h max n). Thus inally it expands some optimal node i h at h, and thereore the solution returned by SOPC satisies: vi ) vi h) v δ h and inally: δ h 1 [ { } ] 3 ) 1 n min CM,1 1 n 1/6 1 min { 1 CM,1})) which is the desired result. i.e. it is δ h -optimal. Note that here we streamlined the proo o the SOO depth bound rom [18] so as to take advantage o the act that, unlike SOO, SOPC always expands a ull path down to h max n). Proo o Theorem 11: Consider irst m > 1. Since hn) is the smallest or which 8) holds, we have: hn) 1 n > CMh maxn) m h = CMh maxn) mhn) 1 Thereore: h = hn) < 1 log m log [ n) CMh maxn) + 1] < c 3 log n 1 ε or some constant c 3 >, where h max n) = n ε was used. Thus hn) is logarithmic in n and, or large n, smaller than h max n) since the latter is a power o n. Thus, or large n, h = hn) in Lemma 1. Similarly solving 8) or a lower bound on hn), we get: hn) 1 log m log [ n) ] 1 CMh maxn) = 1 [ log n 1 ε log ecm ] log m and plugging this into δ hn) : δ hn) 1 log m 1)1 ε)log n log m [ log n 1 ε log ecm ) ] ) where the elimination o the subtracted term holds due to n being large. By Lemma 1 this is also the nearoptimality o the algorithm. When m = 1, 8) becomes simply CMh max n)hn) + 1) n, leading via h max n) = n 1/3 to hn) 3 n CM 1. Thereore { 3 } { } n h = min CM 1, 3 n 3 1 n min CM,1 1 B Execution time o planning algorithms Fig. B.1 shows the execution time o the planning algorithms evaluated in Sec. 5 o the main paper. These results conirm the expectation that the budget n is the most important actor: the execution time is very close to linear in n, with minor dierences between the algorithms due to their varying overhead o e.g. searching the tree in dierent ways. SOPC is the astest. planning time [s] SOPC, planning time OPC, planning time LP, planning time SOOP, planning time n Fig. B.1. Runtime o one call to the planner, averaged over the 5 steps in the trajectory. C Extension o SOPC to multiple actions Although our analysis does not cover it, we briely illustrate an extension o SOPC to control actions. Here, each time step k is associated with intervals rather than just 1. The idea is simple: boxes and steps k are selected or expansion using the same rules as in SOPC, but when a box must be expanded along step k, we split both intervals into M pieces, leading to M = 9 child boxes. More reined rules could be given that avoid this direct exponential growth with the number o inputs. To evaluate this extension, we use the rotational pendulum system rom Sec. 5 o the main paper, but a second motor is added to the vertical joint o the pendulum. This motor has the same viscous damping and torque constant as the irst one, but its model is simpliied in that the torque is linearly related to the voltage u [ 9,9] V, rather than dynamically like or the irst motor. The unnormalized reward unction is x diag[.5,.5,1,.5]x u diag[.,.]u. The C++ implementation o SOPC is used with a budget n = and ε =.33 in h max. The resulting trajectory rom the pointing-down state is shown in Fig. C.1. The angles are approximately stabilized, although with worse perormance and larger input adjustments than

5 r α, α 5 5 α [rad] α [rad/s] θ [rad] θ [rad/s] θ, θ u u 1 [V] u [V] k T s Fig. C.1. Two-input trajectory. in the single-input case, since this simple extension has exponential complexity in the number o actions and we increased the budget less than quadratically).