Bridging the Gap Between Multigrid, Hierarchical, and Receding-Horizon Control

Transcription

1 Bridging the Gap Between Mltigrid, Hierarchical, and Receding-Horizon Control Victor M. Zavala Mathematics and Compter Science Division Argonne National Laboratory Argonne, IL USA ( Abstract: We analyze the strctre of the Eler-Lagrange conditions of a lifted long-horizon optimal control problem. The analysis reveals that the conditions can be solved by sing bloc Gass-Seidel schemes and we prove that sch schemes can be implemented by solving seqences of short-horizon problems. The analysis also reveals that a receding-horizon control scheme is eqivalent to perforg a single Gass-Seidel sweep. We also derive a strategy that ses adjoint information from a coarse long-horizon problem to correct the receding-horizon scheme and we observe that this strategy can be interpreted as a hierarchical control architectre in which a high-level controller transfers long-horizon information to a low-level, short-horizon controller. Or reslts bridge the gap between mltigrid, hierarchical, and receding-horizon control. Keywords: Eler-Lagrange, Gass-Seidel, mltigrid, receding-horizon, hierarchical control 1. BASIC NOTATION AND SETTING We consider the following long-horizon optimal control problem: ( ) T 0 ϕ(z(τ), (τ))dτ ż(τ) = f(z(τ), (τ)), τ [0, T ] z(0) = z. (1a) (1b) (1c) Here, z( ) are the states, ( ) are the controls, and the mappings ϕ( ) and f( ) are assmed to be smooth. We lift the long-horizon problem by partitioning the horizon T into n stages. This lifting approach was proposed by Boc and Plitt (1984) in the context of mltiple-shooting. We define the sets N := {0..n 1} and N := N \ {n 1}; and we assme the stages to be of eqal length h := T/n. The partitioning gives rise to the lifted problem, ( ) N h 0 ϕ(z (τ), (τ))dτ ż (τ) = f(z (τ), (τ)), N, τ [0, h] z +1 (0) = z (h), N z 0 (0) = z. (2a) (2b) (2c) (2d) To simplify or analysis, we transcribe the lifted problem into a finite-dimensional nonlinear programg problem by applying an implicit Eler scheme with m inner stages of eqal length δ := h/m. We define the sets of inner points M := {0..m 1}. We ths obtain the discretized long-horizon problem, 1 Preprint ANL/MCS-P ,j N ϕ(z,j+1,,j+1 ) (3a) (ν,j+1 ) z,j+1 = z,j + δf(z,j+1,,j+1 ), N, j M (3b) (λ ) z,0 = z 1,m, N. (3c) Here, ν,j are the dal variables of the inner dynamic eqations (3b), and λ are the dal variables of the transition eqations between stages (3c). The dal variables are scaled by the constant 1/δ. We se the dmmy variable z 1,m := z to simplify notation. We denote the discretized long-horizon problem (3) as P. Despite advances in comptational methods for optimal control, the long-horizon problem (3) can be difficlt or impossible to solve in real time (reviews on the topic are presented by Diehl et al. (9) and Zavala and Biegler (9)). Traditionally, long-horizon complexity is addressed by sing a receding-horizon scheme. In particlar, one can solve the following short-horizon problems seqentially for = 0,..., N 1:,j ϕ(z,j+1,,j+1 ) (4a) (ν,j+1 ) z,j+1 = z,j + δf(z,j+1,,j+1 ), j M (4b) (λ ) z,0 = z l 1,m. (4c) Here, z 1,m l is fixed and is obtained from the soltion of the problem at 1. We will show that this recedinghorizon scheme is a bloc Gass-Seidel iteration applied to the soltion of the Eler-Lagrange conditions of (3). This observation will help s derive strategies to modify the short-horizon problem (4) so that the receding-horizon scheme better approximates the soltion of (3).

2 2. STRUCTURE OF EULER-LAGRANGE CONDITIONS We grop variables by stages by defining the vectors z := (z,0,..., z,m ), := (,1,...,,m ), and ν := (ν,1,..., ν,m ). We ths obtain the bloc form of P, φ(z, ) N (5a) (ν ) 0 = χ(z, ), N (5b) (λ ) Π z = Π z 1, N, (5c) where the strctre of the mappings φ( ) and χ( ) are given by: φ(z, ) := ϕ(z,j+1,,j+1 ) (6a) z,1 z,0 δf(z,1,,1 ) χ(z, ) :=.. z,m z,m 1 δf(z,m,,m ) (6b) The coefficient matrices Π and Π satisfy Π z = z,0 and Π z 1 = z 1,m. To enable compact notation, we also define the fixed dmmy vector z 1 satisfying Π 0 z 1 = z 1,m = z. The Lagrange fnction of P is given by L(z,, ν, λ ) := φ(z, ) ν T χ(z, ) λ T (Π z Π z 1 ), (7) N and its first-order optimality conditions are 0 = z φ z χ T ν Π T λ + Π T +1λ +1, N (8a) 0 = z φ n 1 z χ T n 1ν n 1 Π T n 1λ n 1 (8b) 0 = φ χ T ν, N (8c) 0 = χ(z, ), N (8d) 0 = Π z Π z 1, N. (8e) Here, z φ := z φ( ), φ := φ( ), z χ := z χ( ), and χ := χ( ). System (8) is the discretetime version of the Eler-Lagrange conditions of the lifted problem (2). Moreover, the dal variables ν,j, λ can be tied together to form discrete-time profiles of the adjoint variables of the lifted problem. These properties are discssed in the boo of Biegler (2010). We consider the following compact form of the Eler- Lagrange conditions (8), F (z 1,m, z,m, λ, λ +1 ) = 0, N (9a) F n 1 (z n 2,m, λ n 1 ) = 0. (9b) Here, we have dropped the dependence on the noncopling variables between stages from the notation. We can see that copling between stages 1,, and + 1 is introdced only throgh z 1,m and λ +1. For clarity, we write F (z 1,m, z,m, λ, λ +1 ) = 0 explicitly: 0 = z φ z χ T ν Π T λ + Π T +1 λ +1 = 0 (10a) 0 = φ χ T ν (10b) 0 = χ(z, ) (10c) 0 = Π z Π z 1 (10d) for N. For F n 1 (z n 2,m, λ n 1 ) = 0 we have 0 = z φ n 1 z χ T n 1ν n 1 Π T n 1λ n 1 = 0 (11a) 0 = φ n 1 χ T n 1ν n 1 (11b) 0 = χ(z n 1, n 1 ) (11c) 0 = Π n 1 z n 1 Π n 1 z n 2. (11d) The bloc strctre of the Eler-Lagrange conditions is not affected by the presence of ineqality constraints and algebraic states and eqations. The concepts presented next apply to this more general setting as well. 3. BLOCK GAUSS-SEIDEL SCHEMES Or ey observation is that we can solve the Eler- Lagrange conditions (9) (or eqivalently (8)) of the long-horizon problem sing bloc, nonlinear Gass-Seidel schemes. Assme that the adjoints λ are fixed to λ l = 0 for N. At = 0 and with fixed z 1 l = z we solve the following optimal control problem: z,j,,j ϕ(z,j+1,,j+1 ) + δ(λ l +1) T z,m (12a) (ν,j ) z,j+1 = z,j + δf(z,j+1,,j+1 ), j M (12b) (λ ) z,0 = z 1,m. l (12c) We refer to this problem as P and introdce the notation,m, λl+1 ) P (z 1,m l, λl +1 ) to indicate the inpts and otpts of problem P. The primal-dal soltion of P solves bloc of the Eler-Lagrange conditions (10) for fixed initial state z 1,m l and adjoint λ +1 = λ l +1. From the soltion of P we obtain the teral state z l+1,m and we se this as initial state for P +1 to compte (z+1,n l, λl +1 ) P,m, λl +2 ). We contine the recrsion ntil reaching the last stage, = n 1. At this stage we solve problem P n 1 : ϕ(z n 1,j+1, n 1,j+1 ) (13a) z n 1,j, n 1,j (ν n 1,j ) z n 1,j+1 = z n 1,j + δf(z n 1,j+1, n 1,j+1 ), j M (13b) (λ n 1 ) z n 1,0 = zn 2,m. l (13c) The primal-dal soltion of P n 1 solves the optimality system (11) for fixed initial state zn 2,m l obtained from the soltion of P n 2. With this step we have pdated all the state (primal) z l+1 and adjoint λ l+1 variables. We retrn to the first stage = 0 and repeat the recrsion to obtain z l+2, λ l+2. We repeat this procedre n GS times. We smmarize the Gass-Seidel scheme below. Gass-Seidel Scheme A I) GIVEN z, set conter l 0, set z l 1,m z, and set λ l 0 for = 0,..., n 1. FOR l = 0,..., n GS DO: II) FOR = 0,..., n 2 SOLVE,m, λl+1 ) P 1,m, λl +1 ). III) FOR = n 1 SOLVE n 1,m, λl+1 n 1 ) P n 1(z l n 2,m, 0).

3 IV) SET l l + 1 and RETURN TO Step II). A ey observation that we mae is that the Gass- Seidel scheme can be implemented by sing off-the-shelf optimization tools. This is becase all that is needed is the ability to solve the nonlinear programg problems P. In particlar, no internal linear algebra maniplations are needed. The strctre of the bloc Gass-Seidel scheme also reveals that a receding-horizon control scheme is eqivalent to perforg a single Gass-Seidel sweep (iteration) with adjoints λ l = 0, N. The adjoints λl +1 encode important global information of the ftre horizon beyond h for problem P. In particlar, the adjoints can be interpreted as teral costs and the term (λ l +1 )T z,m can be seen as a cost-to-go. This information is neglected by the recedinghorizon scheme and this can lead to a poor approximation of the long-horizon soltion. A ey insight that we gain from or analysis is that we can correct the recedinghorizon scheme to better approximate the long-horizon soltion if we are capable of obtaining estimates of the adjoints λ l. Moreover, in the ideal case where or adjoint estimates are optimal, the corrected receding-horizon scheme (a Gass-Seidel sweep) will deliver the optimal long-horizon state profiles. We can obtain refined estimates of the adjoints λ l by perforg mltiple Gass-Seidel iterations of the scheme previosly discssed. This can be seen as a self-correcting receding-horizon scheme. Gass-Seidel pdates can be performed in different ways so we have some degree of flexibility. For instance, Borzı (3) proposed the following forward-bacward scheme: Gass-Seidel Scheme B I) GIVEN z, set conter l 0, set z 1,m l z, and set λ l 0 for = 0,..., n 1. FOR l = 0,..., n GS DO: Forward Sweep II) FOR = 0,..., n 2 SOLVE,m, ) P 1,m, λl +1 ). III) FOR = n 1 SOLVE (zn 1,m l+1, λl+1 n 1 ) P n 1(zn 2,m, l 0). Bacward Sweep IV) FOR = n 2, n 3,..., 0 SOLVE (, λ l+1 ) P 1,m, λl +1 ). V) SET l l + 1 and RETURN TO Step II). In Gass-Seidel Scheme B, the forward sweep pdates the states while the bacward sweep pdates the adjoints. In Gass-Seidel Scheme A the states and adjoints are pdated simltaneosly. Forward-bacward schemes are also typically sed in the soltion of continos-time Eler- Lagrange conditions. For a description of these approaches the reader is referred to the boo of Bryson and Ho (1975). 4. COARSENING-BASED CORRECTION Gass-Seidel (receding-horizon) schemes provide the comptational advantage that they need to solve only shorthorizon problems in order to approximate the soltion of the long-horizon problem. However, Gass-Seidel schemes are well nown for exhibiting slow convergence or no convergence at all. We propose to aid convergence by P c ` +1 m c +1 P m ` +1 ` n 1 n 1 Fig. 1. Transfer of adjoint information from coarse longhorizon problem to short-horizon problems. sing adjoint estimates λ l obtained from the soltion of a coarsened long-horizon problem to correct the shorthorizon problems. Coarsening is a standard concept sed in mltigrid optimal control of partial differential eqations (PDEs). Sch concepts are discssed extensively by Borzì and Schlz (9). To perform coarsening, we consider a coarse grid with m c elements and m c m (ths redcing comptational complexity). We define the coarse set as M c := {0..m c 1} and the corresponding coarsened long-horizon problem P c. We se the notation (λ l 0,..., λ l n 1) P c ( z) to indicate the inpts and otpts of P c. We consider the following scheme: Corrected Gass-Seidel Scheme A I) GIVEN z, set conter l 0, set z l 1,m z. FOR l = 0,..., n GS DO: II) SOLVE (λ l 0,..., λ l n 1) P c ( z). III) FOR = 0,..., n 2 SOLVE,m, λl+1 ) P 1,m, λl +1 ). IV) FOR = n 1 SOLVE n 1,m, λl+1 n 1 ) P n 1(z l n 2,m, 0). V) SET l l + 1 and RETURN TO Step III). In this scheme, P c transfers global (long-horizon) information of problem P to the local (short-horizon) problems P. This is depicted in Figre 1. This approach can be seen as a hierarchical control scheme where a coarsegrained high-level controller spervises a fine-grained lowlevel controller. In other words, the coarse long-horizon problem provides teral costs to the receding-horizon controller so that this better approximates the soltion of the long-horizon problem. In this hierarchical controller dal information is transferred; as opposed to traditional hierarchical control schemes that transfer state information. For a review on hierarchical control see the paper by Scattolini (9). The lifting approach proposed in this wor avoids the need to perform interpolation of state and adjoint profiles in order to move from a coarse grid to a fine grid (as is typically done in mltigrid control for PDEs). In or approach, all that is needed from the coarse problem are the adjoints λ l at the stage transition points. While or analysis borrows concepts of mltigrid control of PDEs sch as Gass-Seidel recrsions and coarsening, or contribtions are the following: We demonstrate how to se mltigrid concepts in a more general setting that might involve constraints and sets of differential and algebraic eqations.

4 We establish a connection between receding-horizon control and Gass-Seidel schemes. We demonstrate that coarsening schemes can be sed to constrct hierarchical control schemes. We demonstrate how to implement mltigrid concepts by sing off-the-shelf optimization tools. 5. NUMERICAL STUDY We illstrate the concepts sing the well-stdied nonlinear CSTR reactor problem: T 0 α c (c(τ) ĉ) 2 + α t (t(τ) ˆt) 2 + α ((τ) û) 2 dτ ċ = 1 c(τ) θ ṫ = t f t(τ) θ ( p exp p E + p exp ( p E t(τ) (14a) ) c(τ) (14b) t(τ) ) c(τ) p α (τ) (t(τ) t c ) (14c) 0 c(τ) 1, 0 t(τ) 1, (τ) 500 (14d) c(0) = c, t(0) = t. (14e) The system states are the concentration of reactant c( ) and the temperatre of reacting mixtre t( ). The control is the cooling water flow ( ). After lifting, we denote the adjoint associated with (14b) as λ c and the adjoint associated with (14c) as λ t. The model parameters are given by α c, α t, α, p α, p E, t f, t c, and p and can be fond in the wor of Zavala and Anitesc (2010). The objective fnction is of Bolza type with the desired targets ĉ, ˆt and û. We highlight the nonlinearity of the system and the presence of bonds. All optimization problems were implemented on AMPL and solved with IPOPT. We partition the time horizon in n = 10 stages and discretize each stage sing an implicit Eler scheme with m = 10 grid points. In Figres 2 and 3 we present the control and temperatre profiles for Gass-Seidel Scheme A. As can be seen, the profiles obtained with a standard receding-horizon scheme (first iteration of Gass- Seidel scheme) severely deviates from the optimal ones. The Gass-Seidel scheme approximates the long-horizon soltion well after three iterations. In Figre 4 we plot the error of the Gass-Seidel Schemes A and B. The error is defined as the Eclidean norm of the error profiles for the states and controls. As can be seen, both schemes have similar performance and converge to error levels of This reslt is srprising since one wold expect that perforg an additional bacward sweep in Scheme B wold be beneficial. Despite the nonlinearity and presence of bonds, both Gass-Seidel Schemes converge to the optimal profiles. In Figre 5 we present the optimal and approximate adjoint profiles obtained from coarsening. The coarse profiles are obtained by discretizing the stage sing m c = 2 points (compared to the m = 10 points sed for the optimal profile). As can be seen, the adjoint profiles have exhibit some errors bt the overall strctre is preserved. We se the coarse adjoint profiles to correct the Gass-Seidel scheme Gass Seidel Optimal Fig. 2. Gass-Seidel A convergence for control. First iteration (top) and third iteration (bottom). t t Gass Seidel Optimal 0.7 Fig. 3. Gass-Seidel A convergence for temperatre. First iteration (top) and third iteration (bottom).

5 Gass Seidel A Gass Seidel B 10 1 Error Iteration Fig. 4. Convergence of Gass-Seidel schemes A and B λ t Optimal Uncorrected Corrected 10 5 Fig. 6. Convergence of Gass-Seidel Scheme A with and withot correction. First iteration (top) and second iteration (bottom) Uncorrected Corrected (m c =2) Corrected (m c =4) λ c Optimal (m=10) Coarse (m c =2) 5 Error Fig. 5. Optimal and coarsened adjoint profiles. A. In Figre 6 we present the convergence of Gass-Seidel with and withot correction. The top graph presents the profiles after one iteration and the bottom graph after two iterations. As can be seen, correction dramatically aids convergence. In Figre 7 we present the convergence of the corrected and ncorrected Gass-Seidel Scheme A. For the corrected case we consider an additional case in which we refine the coarse grid by sing m c = 4 points. As can be seen, the initial error is redced by two orders of magnitde in both cases. 6. CONCLUSIONS AND FUTURE WORK We presented an analysis of the Eler-Lagrange conditions for a lifted optimal control problem. This enabled s to derive bloc Gass-Seidel schemes that enable the soltion of long-horizon problems by solving seqences of short-horizon problems. Or analysis revealed that Iteration Fig. 7. Convergence of Gass-Seidel Scheme A with and withot correction. a receding-horizon scheme is eqivalent to perforg a Gass-Seidel sweep of the Eler-Lagrange conditions. We have also sed or analysis to derive strategies to correct adjoint profiles by sing coarsening. This approach enabled s to accelerate Gass-Seidel convergence and can be interpreted as a hierarchical control strctre in which a coarse high-level controller transfers long-horizon information to a low-level, short-horizon controller. Or reslts ths bridge the gap between mltigrid, hierarchical, and receding-horizon control. As part of ftre wor, we wold lie to gain additional insight on conditions garanteeing convergence of Gass-Seidel schemes and we will se mltigrid control concepts to design alternative control architectres.

6 ACKNOWLEDGEMENTS This material is based pon wor spported by the U.S. Department of Energy, Office of Science, nder contract nmber DE-AC02-06CH The athor thans Mihai Anitesc for pointing ot existing wor on mltigrid control. REFERENCES Biegler, L.T. (2010). Nonlinear programg: Concepts, algorithms, and applications to chemical processes. SIAM. Boc, H.G. and Plitt, K.J. (1984). A mltiple shooting algorithm for direct soltion of optimal control problems. 9th IFAC World Congress, Bdapest. Borzı, A. (3). Mltigrid methods for parabolic distribted optimal control problems. Jornal of Comptational and Applied Mathematics, 157(2), Borzì, A. and Schlz, V. (9). Mltigrid methods for PDE optimization. SIAM Review, 51(2), Bryson, A.E. and Ho, Y.C. (1975). Applied Optimal Control. Hemisphere. Diehl, M., Ferrea, H.J., and Haverbee, N. (9). Efficient nmerical methods for nonlinear MPC and moving horizon estimation. In Nonlinear Model Predictive Control, Scattolini, R. (9). Architectres for distribted and hierarchical model predictive control A review. Jornal of Process Control, 19(5), Zavala, V.M. and Biegler, L.T. (9). Nonlinear programg strategies for state estimation and model predictive control. In Nonlinear Model Predictive Control, Zavala, V.M. and Anitesc, M. (2010). Real-time nonlinear optimization as a generalized eqation. SIAM Jornal on Control and Optimization, 48(8), The sbmitted manscript has been created by UChicago Argonne, LLC, Operator of Argonne National Laboratory ( Argonne ). Argonne, a U.S. Department of Energy Office of Science laboratory, is operated nder Contract No. DE-AC02-06CH The U.S. Government retains for itself, and others acting on its behalf, a paid-p, nonexclsive, irrevocable worldwide license in said article to reprodce, prepare derivative wors, distribte copies to the pblic, and perform pblicly and display pblicly, by or on behalf of the Government.