Idempotent Method for Continuous-Time Stochastic Control and Complexity Attenuation

Transcription

1 Idempotent Method for Continuous-ime Stochastic Control and Complexity Attenuation illiam M. McEneaney Hidehiro Kaise Seung Hak Han Dept. of Mech. and Aero. Eng., University of California San Diego, San Diego, CA , USA ( Graduate School of Information Sci., Nagoya University, Furo-cho, Chikusa-ku, Nagoya , Japan Dept. of Mech. and Aero. Eng., University of California San Diego, San Diego, CA , USA ( Abstract: e consider a -plus based numerical method for solution of finite timehorizon control of nonlinear diffusion processes. he approach belongs to the class of curseof-dimensionality-free methods. he -plus distributive property is required. he price to pay is a very heavy curse-of-complexity. hese methods perform well due to the complexityattenuation step. his projects the solution down onto a near-optimal -plus subspace. Keywords: stochastic, nonlinear control, numerical methods. 1. INRODUCION It is now well-known that many classes of deteristic control problems may be solved by max-plus or plus numerical methods. hese methods include maxplus basis-expansion approaches [1], [5], [7], as well as the more recently developed curse-of-dimensionality-free methods [7], [10]. It has recently been discovered that idempotent methods are applicable to stochastic control and games. he methods are related to the above curseof-dimensionality-free methods for deteristic control. In particular, a -plus based method was developed for stochastic control problems [9]. he first such methods for stochastic control were developed only for discrete-time problems. Here, we will remove the severe restriction to discrete-time problems. his extension requires overcog significant technical hurdles. e will first define a parameterized set of operators, approximating the dynamic programg operator. e obtain the solutions to the problem of backward propagation by repeated application of the approximating operators. Using techniques from the theory of viscosity solutions, we show that the solutions converge to the viscosity solution of the Hamilton-Jacobi-Bellman partial differential equation (HJB PDE) associated with the original problem. he problem is now reduced to backward propagation by these approximating operators. he -plus distributive property is employed. A generalization of this distributive property, applicable to continuum versions will be obtained. his will allow interchange of expectation over normal random variables (and other random variables with range in IR m ) with infimum operators. At each time-step, Research partially supported by NSF Grant DMS , AFOSR Grant FA and Grant-in-Aid for Young Scientists (B), No , MEX. the solution will be represented as an infimum over a set of quadratic forms. Use of the -plus distributive property will allow us to maintain that solution form as one propagates backward in time. Backward propagation is reduced to simple standard-sense linear algebraic operations for the coefficients in the representation. e also demonstrate that the assumptions on the representation which allow one to propagate backward one step are inherited by the representation at the next step. he difficulty with the approach is an extreme curse-of-complexity, wherein the number of terms in the -plus expansion grows very rapidly as one propagates. he complexity growth will be attenuated via projection onto a lower dimensional plus subspace at each time step. hat is, at each step, one desires to project onto the optimal subspace relative to the solution approximation. Importantly, the subspace is not set a priori. Using some tools from convex analysis and imax theory, we show the optimal projection is achieved by pruning the current set of quadratic forms.. DEFINIION AND DYNAMIC PROGRAM e begin by defining the specific class of problems which will be addressed here. Let the dynamics take the form dξ s = f µs (ξ s, u s ) ds + σ µs (ξ s, u s ) db s, ξ t = x IR n, (1) where f m (x, u) is measurable, with more assumptions on it to follow. he u s and µ s will be control inputs taking values in U IR p and M =]1, M[ = 1,,, M, respectively. In practice, we often find it useful to allow both a continuum-valued control component and a finite set-valued component, where the latter is used to allow approximation of more general nonlinear Hamiltonians, c.f. [7] for motivation. Also, B, F is an l-dimensional Brownian motion on the probability space (Ω, F, P ), where F 0 contains all the P -negligible elements of F and σ m (x, u) is an n l matrix-valued diffusion coefficient. e will be Copyright by the International Federation of Automatic Control (IFAC) 316

2 exaing a finite time-horizon formulation, with teral time,, and will take initial time t [0, ]. he payoff (to be imized) will be J(t, x, u, µ ) =E. l µs (ξ s, u s ) ds+ Ψ(ξ ) t where Ψ(x) =. inf z Z g (x, z ), (3) where l m (x, u) and g (x, z) are measurable, and (Z, d Z ) is a separable metric space. he value function is V (t, x) = inf J(t, x, u, µ ), u U t,µ M t (4) where U t (resp. M t ) is the set of F t -progressively measurable controls, taking values in U (resp. M) such that there exists a strong solution to (1). () e will assume that the given data in the dynamics and payoff satisfy the following conditions: (A1) U is a closed subset of IR p. (A) here exist L f, L σ, K f, K σ > 0 such that for any x, x IR n, u, u U, m, m M, f m (x, u) f m (x, u ) L f ( x x + u u + m m ), σ m (x, u) σ m (x, u ) L σ ( x x + u u + m m ), f m (x, u) K f (1 + x + u ), σ m (x, u) K σ. (A3) here exist L l, K l > 0 and ν > 0 such that for any x, x IR n, u, u U, m, m M l m (x, u) l m (x, u ) L l (1 + x + x + u + u ) ( x x + u u + m m ), K l + ν u l m (x, u) K l (1 + x + u ). (A4) here exist L Ψ, K Ψ > 0 such that for any x, x IR n, Ψ(x) Ψ(x ) L Ψ (1 + x + x ) x x, K Ψ Ψ(x) K Ψ (1 + x ). At a formal level, one expects that V (t, x) will satisfy the following Hamilton-Jacobi-Bellman partial differential equation (HJB PDE): V t (t, x) + H(x, D xv (t, x), D xv (t, x)) = 0, (5) V (, x) = Ψ(x), x IR n, (6) H(x, D x V (t, x), DxV (t, x)) 1 = inf u U m M tr(σm (x, u)σ m (x, u) DxV (t, x)) +f m (x, u) D x V (t, x) + l m (x, u). (7) Under restrictive conditions, it is known that V (t, x) can be characterized as the unique viscosity solution of (5)/(6). However, for more general conditions including assumptions (A1) (A4), if V (t, x) is[ associated with the ] set of control processes satisfying E u t s ds <, it is shown in [4] that V (t, x) is the unique viscosity solution of (5)/(6) satisfying the quadratic growth condition V (t, x) K(1 + x ), (t, x) [0, ] IR n. (8) See [4, heorem 3.1 and Remark 3.1]. e need to additionally show convergence of discrete-time approximations to the viscosity solution. e approximate the viscosity solution of (5)/(6) by discrete-time stochastic control problems. o define a discrete-time stochastic control value, we introduce a family of parameterized operators F t,s t<s acting on φ : IR n IR: F t,s φ(x) = inf u U,m M lm (x, u)(s t) (9) +E[φ(x + f m (x, u)(s t) + σ m (x, u)(b s B t ))]. Let π N = t 0 = 0 < t 1 < < t N = be a partition of [0, ] with the step size N = t i+1 t i = /N (i = 0, 1,, N 1). e define a discrete-time value function V N (t, x) ((t, x) [0, ] IR n ) associated with π N recursively backward in time: Ψ(x), t =, V N (t, x) = F t,ti+1 V N (10) (t i+1, )(x), t i t < t i+1, where F t,ti+1 V N (t i+1, )(x) is F t,ti+1 φ(x) with φ( ) = V N (t i+1, ). e obtain the following theorem regarding approximation of the viscosity solution. heorem.1. Under (A1) (A4), V N (t, x) converges to a continuous viscosity solution V (t, x) of (5) with (6) as N uniformly on each compact set of [0, ] IR n. Indeed, V (t, x) is the unique viscosity solution satisfying the quadratic growth condition: for some K > 0, V (t, x) K(1 + x ), (t, x) [0, ] IR n. (11) 3. MIN-PLUS DISRIBUIVE PROPERY e will use an infinite version of the -plus distributive property to move a certain infimum from inside an expectation operator to outside. It will be familiar to control and game theorists who often work with notions of nonanticipative mappings and strategies. One version of such appeared in [9]. However, the assumptions in that result are too restrictive for the class of problems we are considering. Instead, we generalize that result to the following, where the proof appears in [6]. heorem 3.1. Let (Z, d Z ) be a separable metric space and (, d ) be a separable Banach space with Borel sets B. Let p be a finite measure on (, B ), and let D =. p( ). Let h : Z IR be Borel measurable. Suppose there exists z Z such that h(w, z) dp (w) < (1) and suppose for given ε > 0, there exists R < such that h(w, z) dp (w) ε. (13) (B R (0)) c inf z Z Also, suppose that given ε > 0 and R <, there exists δ > 0 such that h(w, z) h( w, z) < ε for all z Z and all w, w B R (0) such that d (w, w) < δ. Lastly, we suppose that either Z is countable or h(w, z) is continuous on z for each w (where of course, the 317

3 former supposition can be embedded within the latter, but that is less illuating). hen, inf h(w, z) dp (w) = inf h(w, z(w)) dp (w), z Z z Z where Z. = z : Z Borel measurable. 4. DISRIBUED DYNAMIC PROGRAMMING e will use the above infinite-version of the -plus distributive property in conjunction with the dynamic programg principle of Section. his will yield what we refer to as an idempotent distributed dynamic programg principle (IDDPP). Recall our discrete-time value function, V N (t k, x) given by (10) for t k π N and x IR n. Suppose that at time, t k+1, one has representation V N (t k+1, x) = inf g N k+1(x, z), (14) where (Z k+1, d Zk+1 ) is a separable metric space. Recalling form (3), and letting gn N(x, z) = g (x, z) and Z N = Z, we see that V N (t N, x) = V N (, x) = Ψ(x) has this form. hen the dynamic program of (9), (10) with = /N becomes V N (t k, x) = inf l m (x, u) u U m M [ ( +E inf x + f m (x, u) + σ m (x, u)w, z )] g N k+1 = inf u U m M [ inf l m (x, u) +g N k+1( x + f (x, u, m, w), z )] dp (w), (15) where f (x, u, m, w) = f m (x, u) + σ m (x, u)w, P is the measure corresponding to a Gaussian random variable over IR l with mean zero and covariance I, and = IR l. e outline the approach to be followed. Specifically, we will use the -plus distributive property of heorem 3.1 to move the infimum over Z k+1 outside the integral. Letting Z k+1. = zk+1 : Z k+1 Borel measurable, we will have V N (t k, x) = inf u U m M inf z k+1 Z k+1 l m (x, u) (16) +g N k+1(x + f (x, u, m, w), z k+1 (w))dp (w). Since we will suppose that Z k is bounded and U is possibly unbounded, it is convenient to handle the infimum over u separately in (16). From (16), we can take V N (t k, x) = inf z Z k g N k (x, z), (17) where Z k = M Z k+1 and for x IR n and z = (m, z k+1 ) Z k, gk N (x, z) = inf l m (x, u) (18) u U +g N k+1(x + f (x, u, m, w), z k+1 (w))dp (w). Consequently, the general form of (14) will be inherited from V N (t k+1, ) to V N (t k, ), and one can propagate backward in this manner indefinitely. his is what we referred to above as the IDDPP. e note that (16) itself can be used as an IDDPP if U is bounded. ith significant effort, one obtains our main IDDPP result. heorem 4.1. In addition to (A1) (A3), suppose that (Z, d Z ) and g : IR n Z IR satisfy the following: (i-) (Z, d Z ) is a bounded and closed subset of a separable Banach space X where metric d Z is induced by the norm of X. (ii-) here exist K, L > 0 such that for any x, x IR n, z Z, K g (x, z) K(1 + x ), g (x, z) g (x, z) L(1 + x + x ) x x. (iii-) here exists ˆL > 0 such that for any z, z Z, x IR n, g (x, z) g (x, z ) ˆL(1 + x )d Z (z, z ). Letting Z N = Z and gn N (x, z) = g (x, z), (17) with (18) holds for all k ]0, N[. 5. FULLY QUADRAIC FORMS he above theory was somewhat general in form. ith a small modification, one could include u in the index set, letting Z k = U M Z k+1 rather than Z k = M Z k+1. However, we believe the most computationally useful form will occur when the problem is quadratic in u with U = IR p. Consequently, we now consider such a special problem class, where in particular, we use the fact that the quadratic nature of the control-dependence allows one to analytically obtain the g N k for each z Z k. o be precise, we now suppose f m (x, u) = A m x + B m u + λ m d, σ m (x, u) = σ m, (19) l m (x, u) = 1 x H m 1 x + x H m u + 1 u H m 3 u + λ m 1 x + λ m u + c m, U = IR p, (0) g (x, z) = 1 x Q (z)x + b (z)x + c (z). (1) e do not use the earlier assumptions. Instead, we replace them with assumptions on the problem data in the above form which will yield a special case of the earlier assumptions. e assume: (AQ.1) (Z, d Z ) is a bounded and closed subset of a separable Banach space X where metric d Z is induced by norm X of X. (AQ.) here exists D 1 < such that Q (z), b (z), c (z) D 1 for all z Z. (AQ.3) here exists L < such that max Q (z) Q ( z), b (z) b ( z), c (z) c ( z) Ld Z (z, z) z, z Z. (AQ.4) Q (z) 0 for all z Z. 318

4 (AQ.5) H m 3 > 0 and ( H m 1 H m ) 0 m M. H m H m 3 Note that (AQ.1) (AQ.5), along with the specific form of (19) (1) immediately imply that (A1) (A4) and (i-) (iii-) hold. Recall (18), and suppose that for some k ]0, N 1[, Vk+1(x) N =. V N (t k+1, x) = inf gk+1(x, N z), () where g N k+1(x, z) = 1 x Q k+1 (z)x + b k+1(z)x + c k+1 (z). (3) Note that g N N (x, z) = g (x, z) is of this form. After some computations, we arrive at the following, where we again do not include the details of the algebra. g N k (x, z)= 1 x Q k ( z, m)x + b k ( z, m)x + c k ( z, m) = 1 x Q k (z)x + b k (z)x + c k (z) (4) for appropriate Q k, b k, c k, where z = (m, z) Z k. Using this form, one can verify that conditions (AQ.1) (AQ.4) are inherited. e do not include the details. his yields the following or, but computationally useful, variant of main heorem 4.1. heorem 5.1. Consider the special dynamics and cost of (19) (1). Suppose (AQ.1) (AQ.5) hold. Let gn N (x, z) = g (x, z) and Z N = Z. hen, for all k ]0, N 1[ Vk N (x) = V N (t k, x) = inf gk N (x, z), z Z k where g N k (x, z) is given by (4) with Z k = Z k+1 M, for appropriate Q k, b k, c c. Due to space limitations, we do not include the computations which yield Q k, b k, c k from Q k+1, b k+1, c k COMPLEXIY REDUCION he key to this class of methods lies in the repeated projection of the solution down onto a low-dimensional (-plus) subspace. Importantly, the subspace is chosen at each step so as to imize the error induced by this projection. One sees from heorem 5.1 that after one step of the IDDPP, the set Z k will have the cardinality of the continuum even in the case where Z k+1 is finite. Consequently, the projection down to a finite-dimensional subspace is a critical step. For the class of problems where the solution may be well-represented by a reasonable set of quadratic functions, this method can be expected to perform quite well. It is worth noting that this complexity condition is independent from the dimension of the space; one may have problems with low complexity (by this vaguely defined metric) solutions in high dimensional spaces and vice-versa. e now begin the analysis of the projection of the V N k down to a low-dimensional -plus subspace. his will be a two-step procedure. First, we indicate a simple reduction from an infinite-dimensional index set, Z k, to a, possibly very large but finite, subset. Second, we discuss the theory by which, given a desired dimension of the subspace (i.e., index set size), one finds the optimal subspace of that dimension. e also indicate a computational approach for approximately achieving this projection. e will work entirely in the special case of Section 5. For the reduction from infinite-dimensional to finitedimensional, we simply note that by (AQ.), there exists D < such that (Q k (z), b k (z), c k (z)) z Z k B 3D (0), which is compact. herefore, given ε > 0, there exists a finite ε-net for the set, which yields our finitedimensional -plus projection. As indicated above, the key to computational feasibility will be projection of V k N down to a low-dimensional plus subspace with small error, if such is possible. e will use an approach similar to that in [8]. hat is, we will devise an optimization problem to select the small set of quadratics whose pointwise imum (-plus sum) will constitute the -plus projection. he constraint in the optimization will be slightly relaxed from the ideal constraint. ith this optimization problem formulation, we will see that our problem reduces to selection of a small set of quadratics out of the set whose -plus sum is V k N. Let be the set of all triples of coefficients for quadratic functions on IR n. hat is, we take = (Q, b, c) Q IR n n, symmetric; b n IR n ; c IR. For convenience, we will designate a triple by a single symbol, say τ, where τ will correspond to some triple, say τ = (Q, b, c). e let C Q (IR n ) denote the subset of C(IR n ) consisting of the quadratic functions mapping into IR. e let G : C Q be given as follows. For τ = (Q, b, c), let G[τ](x) = G[(Q, b, c)](x). = 1 x Qx + b x + c. Consider a finite set of quadratic coefficients, M given by. M = τ M, (5). where M =]1, M[ = 1,,... M with M IN. (Here, of course, each τ corresponds to a triple of coefficients.) Note that M may be very large. In particular, we will be interested in the specific case where M corresponds to the set of quadratics which generate V k N, in which case, M = N ε = #Ẑk. Further, in that case, each triple in M which we now designate as τ = ( Q, ˆb, ĉ ) corresponds (via bijection) to some triple (Q k (Ẑ), bk(ẑ), ck(ẑ)). e look for an optimal smaller set of triples, A N = α n n N, N =]1, N[ where N < M. Let Π be the set of probability measures over (IR n, B n ) where B n is the collection of Borel sets over IR n. Integration with respect to a π Π will be designated by π(dx). Let λ be a measure over (IR n, B n ) such that G[τ](x) λ(dx) < for all τ. he optimization IR problem n will take the form Minimize J(A N) =. IR n n N G[α n ](x) λ(dx) (6) subject to G[α n ](x) π(dx) IR n G[τ ](x) π(dx) IR n π Π, n N. (7) e note that (7) is a relaxation of constraint G[α n ](x) G[τ ](x) x IR n. (8) 319

5 Further, constraint (8) is typical in -plus expressions. If there exists n N, x IR n and ε > 0 such that G[α n ]( x) G[τ ]( x) ε, then addition of more α n to A N can never correct this error as one will still have n N G[α n ]( x) G[τ ]( x) ε. e will analyze problem (6),(7) to show that this formulation corresponds to imization of a monotonically increasing, concave function over a cornice set [8], where this class of problems has the property that the imizing set, A N, will be a subset of the original set, M. hat is, the optimal -plus subspace of dimension N will be a subset of of the original set M. For τ, α, we say α τ if G[α](x) π(dx) G[τ](x) π(dx) π Π. IR n IR n Note that is a partial order on. It is also worth noting the following. Lemma 6.1. α τ if and only if G[α](x) G[τ](x) for all x IR n. Given Lemma 6.1, one might wonder why we do not use the condition G[α](x) G[τ](x) for all x IR n to define our partial order. hen combined with the imization operation as on the right-hand side of (7), these are different; the integral definition will be more useful. For π Π, let G π [τ]. = IR n G[τ](x) π(dx). Lemma 6.. For any π Π, G π is a linear functional on. Let S M. = p IR M p [0, 1] M, p = 1. For any p S M, define p M = p τ. e say M = τ τ = p M for some p S M, (9) which is obviously the convex hull of M. Lemma 6.3. For any π Π, G π [τ ] = G π [p M ], p S M or equivalently, IR n G[τ ](x) π(dx) = p S M IR n G[p M ](x) π(dx). One then immediately has: Lemma 6.4. Fix any π Π. Let α M. hen, G π [α] G π [τ ] if and only if G π [α] p S M G π [p M ]. Using Lemma 6.4, our problem (6),(7) becomes Minimize J(A N) =. IR n n N G[α n ](x) λ(dx) (30) subject to G π [α n ] G π [p M ] p S M π Π, n N. (31) Given, we define the upper cone of as U[ ]. = α α τ for some τ. e also define the cornice of as C[ ] =. U[ ] (see [8]), and let C N [ ] =. C[ ] N denote the outer product of C[ ], N times. heorem 6.5. Condition (31) holds if and only if A N C N [ M ]. Combining heorem 6.5 with (30)/(31), we see that problem (6)/(7) is equivalent to: Minimize J(A N ) =. IR n n N G[α n ](x) λ(dx) (3) subject to A N C N [ M ]. (33) Define the inherited partial order on outer product space [ ] N by A 1 N A N where A j N = α j n α j n n N for j 1,, if α 1 n α n for all n N. [ ] N Lemma 6.6. J : IR is monotonically increasing (relative to ) and concave. Now, given the monotonicity and concavity of J, we can apply [8] heorem 3.1 to assert: heorem 6.7. Let J be the optimal value of problem (3),(33), or equivalently, (6),(7). hen, there exists A N = τ n n N such that τ n M for all n N, and such that A N satisfies (33) (equivalently, (7)) and J(A N) = J. he value of heorem 6.7 is that we know that the optimal set of quadratics is a subset of the original M set. It is worth noting that condition A N C N[ M ] is not identical to the condition G[α n ](x) G[τ ](x) x IR n n N. Instead, it is somewhat more conservative. 6.1 ractable Pruning Now that we understand that pruning is optimal (relative to the chosen criterion and constraints), we consider means for approximate optimal pruning. In particular, solution of even the reduced-complexity pruning problem with the above criterion remains computationally demanding. Consequently, we search instead for a tractable suboptimal pruning method. Firstly, one would like to remove those quadratics that do not contribute at all to the value function. Let 0, and suppose ˆm M is such that + G[τ ˆm ](x)π(dx) G[τ ](x)π(dx), IR n \ ˆm IR n (34) for all π Π. One obtains: Proposition 6.8. Suppose G[τ ˆm ](x)π(dx) G[τ ](x)π(dx), IR n \ ˆm IR n for all π Π. hen, 30

6 and IR n = G[τ ](x) π(dx) IR n \ ˆm G[τ ](x) G[τ ](x) π(dx), π Π = \ ˆm G[τ ](x), x IR n. In other words, τ ˆm does not contribute at all to the pointwise imum, and so may be removed without affecting the value function approximation. Further, by the monotonicity of the semigroup operator, no progeny of τ ˆm contribute to the value function approximation in subsequent steps, and so removal of τ ˆm does not affect the solution at any time-step. Now let. ˆm = inf (34) holds. It is apparent that ˆm provides some measure (loosely used) of the merit of τ ˆm relative to the imum. his leads to the following rough heuristic: o achieve a good subset of size N from the original set of M quadratics, those with the lowest ˆm values should be chosen. his leads to the optimization of a submodular functional, which is an issue we will discuss elsewhere. Here, however, we consider the approximate evaluation of ˆm. 6. Gaussians and Convex Program Given a set of quadratics, (Q, b, c ) M indexed as above by its coefficients, corresponding to some set τ M for some set M M, and a specific quadratic, (Q ˆm, b ˆm, c ˆm ) corresponding to τ ˆm, we wish to compute an approximate ˆm. In order to make the problem tractable, we consider only those π Π corresponding to normal distributions. Let D be the set of positive-definite, symmetric n n matrices. he appropriate set of normal distributions may be indexed by (D, x) D IR n. Let (Q ˆm, b ˆm, c ˆm ) correspond to τ ˆm, and π Π correspond to (D, x) D IR n. hen, G[τ ˆm 1 [ 1 ](x)π(dx)= IR n (π) n/ det(d) IR x Q ˆm x n +b ˆm x + c ˆm] exp 1 (x x) D 1 (x x) dx, and after manipulation, one finds this is = 1 x Q ˆm x + b ˆm x + c ˆm + 1 tr [ Q ˆm D ]. Note that this is linear in D and quadratic in x. Let Π denote the set of normal probability measures over IR n. ith a little work, one finds that ˆm may be computed by solution of: [ 1 max z (D, x,z) D IR n+1 x Q ˆm x + b ˆm x + c ˆm + 1 tr [ Q ˆm D ] ] [ 1 subject to: z x Q x + b x + c (35) + 1 tr [ Q D ]] 0 M. his last problem may be converted to an LMI. Following a similar approach to that used in [10], one obtains a relaxed dual problem: ζ ( ) λ c c ˆm 1 ζ λ b b ˆm ( ) ( ) 1 λ b b ˆm 1 λ Q Q ˆm 0, λ Q Q ˆm 0, (λ,ζ) S M IR where S M =. λ IR M λ m [0, 1] m, m λ m = 1, which takes the form of an LMI. hese LMIs are solved repeatedly to detere the contribution of each quadratic to the overall value function approximation. As noted above, the actual pruning corresponds to optimization of a submodular function on the space of subsets of M, and the details are not included due to length restrictions. REFERENCES [1] M. Akian, S. Gaubert and A. Lakhoua, he maxplus finite element method for solving deteristic optimal control problems: basic properties and convergence analysis, SIAM J. Control and Optim., 47 (008), [] G. Barles and P.E. Souganidis, Convergence of approximation schemes for fully nonlinear second order equations, Asymptotic Anal., 4 (1991), [3] G. Cohen, S. Gaubert and J.-P. Quadrat, Duality and Separation heorems in Idempotent Semimodules, Linear Algebra and Applications, 379 (004), [4] F. Da Lio and O. Ley, Uniqueness results for secondorder Bellman-Isaacs equations under quadratic growth assumptions and applications, SIAM J. Control Optim., 40 (006), [5].H. Fleg and.m. McEneaney, A max-plus based algorithm for an HJB equation of nonlinear filtering, SIAM J. Control and Optim., 38 (000), [6] H. Kaise and.m. McEneaney, Idempotent Expansions for Continuous-ime Stochastic Control, Proc. 49th IEEE Conf. Decision and Control, (010). [7].M. McEneaney, Max-Plus Methods for Nonlinear Control and Estimation, Birkhauser, Boston, 006. [8].M. McEneaney, Complexity Reduction, Cornices and Pruning, Proc. of the International Conference on ropical and Idempotent Mathematics, G.L. Litvinov and S.N. Sergeev (Eds.), Contemporary Math. 495, Amer. Math. Soc. (009), [9].M. McEneaney, Idempotent Algorithms for Discrete-ime Stochastic Control, Automatica (to appear). [10].M. McEneaney, A. Deshpande, S. Gaubert, Curse-of-Complexity Attenuation in the Curse-of- Dimensionality-Free Method for HJB PDEs, Proc. ACC 008, Seattle (008). 31