Static and Dynamic Optimization (42111)

Transcription

1 Static and Dynamic Optimization (42111) Niels Kjølstad Poulsen Build. 303b, room 016 Section for Dynamical Systems Dept. of Applied Mathematics and Computer Science The Technical University of Denmark phone: mobile: Lecture 7: Free dynamic optimization (D) 1/48

2 Outline of lecture What is Dynamic Optimization Dynamics - examples Cost functions - examples Free dynamics optimization Euler-Lagrange equations Example: optimal stepping Practicalities Exercise DO.1 Reading: DO: 5-11, /48

3 Dynamic Optimization What is Dynamic Optimization? Dynamic Optimization has 3 ingredients: Some dynamics. Here (in this course) a state space model. A performance index (cost function, objective function). In our case it is a summation (or integral) of contribution over a period of time of fixed or free length (part of the optimization). Eventually some constraints (on the decisions or on the states) 3/48

4 Problem statement (Free dynamic (D) optimization) Task: Find a sequence of optimal decisions (u i i = 0,... N 1). Dynamics (described by a state space model): x i+1 = f i (x i,u i ) i = 0, 1,... N 1 x 0 = x 0 Objective: (to minimize or maximize the index): N 1 J = φ N (x N )+ L i (x i,u i ) i=0 Here N and x 0 are fixed (given), J, φ and L are scalars. x i and f i are n-dimensional vector and vector function and u i is a m-dimensional vector of decisions. Notice: no constraints (except given by the dynamics) i.e. Free Dynamic Optimization. 4/48

5 Ex1: Optimal Pricing (simplified) We are producing a product (brand A) and have to determine its price in order to maximize our income. There is a competitor product B - and a problem. If we are to modest we might have almost all the costumers - but we will not earn that much. If we are to greedy then the bulk majority of the costumers will bye the other brand B - and we will lose money. We devide the period into N intervals and going for a price profile (u i, i = 0,... N 1). 5/48

6 Optimal Pricing - the performance index We have to decide the price of the product u i u (u being the production cost) in each interval N Let M be the size of the marked and let x i (0 x i 1) be the (A) share of the marked in the i th interval. Objective: to make some money - i.e. to maximize N 1 ( Max J where J = M x i ui u ) x i = 1 2 (x i +x i+1 ) i=0 More precisely, x i is the marked share at the beginning of interval i and x i is the average share of the marked in interval i (i = 0,... N 1). 6/48

7 Optimal Pricing - the dynamics A q 1 x B x p p Transition probability A >B q Transition probability B >A 1 Escape prob. 1 Attraction prob. 0 A > B 0 B > A price price Dynamics: A A B A x i+1 = ( 1 p[u i ] ) x i +q[u i ] ( 1 x i ) x 0 = x 0 7/48

8 Recap Optimal Pricing Dynamics: x i+1 = ( 1 p(u i ) ) x i +q(u i ) ( 1 x i ) Objective: x 0 = x 0 N 1 ( Max J where J = M x i ui u ) i=0 Notice: This is a discrete time model. No constraints. The length of the period (the horizon, N) is fixed. 8/48

9 If u i = u+5 (u = 6, N = 10) we get J = 8 (rounded to integer) x u /48

10 Optimal pricing (given correct model): J = 27 (rounded to integer) x u Notice different axis for x. 10/48

11 Free Dynamic Optimization Dynamics (described by a state space model): x i+1 = f i (x i,u i ) x 0 = x 0 Objective (to optimize the index): N 1 J = φ N (x N )+ L i (x i,u i ) i=0 Here N and x 0 are fixed (given), J, φ and L are scalars. x i and f i are n-dimensional vector and vector function and u i is a vector of decisions. Notice: no constraints (except given by the dynamics). 11/48

12 Example: Optimal pricing In this case: The dynamic is given by: f i (x i,u i ) = ( 1 p(u i ) ) x i +q(u i ) ( ) 1 x i where x 0 = 0 and N = 10. The objective function (performance index) is characterized by: L i (x i,u i ) = M x i ( ui u ) φ N (x N ) = 0 x i = 1 2 [ xi +x i+1 ] where x i+1 = ( 1 p[u i ] ) x i +q[u i ] ( 1 x i ) 12/48

13 Dynamics Dynamics (described by a state space model): x i+1 = f i (x i,u i ) i = 0, 1,... N 1 x 0 = x N Let us have a look at some other examples: 13/48

14 Example: The Schaefer model Common growth model in connection to biological systems. E.g. (re)production of fish. State x i Biomass (mass of fish) in a given area and at the beginning of interval i. Decision u i Mass of fish to be catch in interval i. Parameters r and α x i+1 = x i +rx i ( 1 αx i ) u i x 0 = x 0 i = 0, 1,... N x i x i = x i+1 x i x i This is a first order (one state), but a non-linear model. Stationarity and Stability. 14/48

15 Example: The Lotka-Volterra model A Prey-Predator (the Lotka-Volterra (1925) model or the rabbit[r]-fox[f]) model). [ ] [ ] [ ] [ ] [ ] ri+1 ri α1 r Dynamics: = + i β1 r i F i ui F i+1 F i β 2 r i F i α 2 F i v i birth death hunt 180 Lotka Volterra rabbit # fox, rabbit fox period A nonlinear, second order model with oscillations. 15/ 48

16 Dynamics Dynamic (discrete time) state space model: x 1 x 1 u x 2 x 2 1 = f i,... ; u x n x m i+1 n i i or in short: x 1 x 2. x n = 0 x 1 x 2.. x n 0 x i+1 = f i (x i,u i ) x 0 = x 0 f : R n+m+1 R n The fox-rabbit example: [ r F ] i+1 [ ] ri +α = 1 r i β 1 r i F i u i F i +β 2 r i F i α 2 F i v i 16/48

17 The State Space concept The state space model: x i+1 = f i (x i,u i ) x 0 = x 0 State x i R n : vector containing quantities describing the situation (the state) of the system. They contain information of the system history (is a sufficient statistics of the history). No historical data is needed for determining future state values (x τ where τ > i) if present state vector x i is known. Notice, this is a Markov property. Input u i R m : vector containing the inputs. Might be control inputs or decision variables. It might be the inputs from an opponent or simply a disturbance. Here (for the time being), we are going to determine a sequence of optimal inputs i /48

18 LTI system Linear Time invariant (LTI) systems x 1 x 2. x n i+1 = A x 1 x 2. x n u 1 +B. u m i i or in short x i+1 = Ax i +Bu i An example: [ x1 x 2 ] i+1 [ 1 0 = ][ x1 x 2 ] i [ ] u i [ x1 x 2 ] 0 [ 0 = 0 ] 18/48

19 Stability A glimpse of stability Common sense definition: States do not grow unlimited. Example 1: Unattended bank loan (r=0.02) : x i+1 = [1+r]x i x 0 = x 0 Example 2: Controlled loan with a down payment equals u i = 0.04x i x i+1 = [1+r]x i u i = 0.98x i x 0 = x /48

20 Stability - the LTI case First order (LTI) system (ie. x i, a are scalars) x i+1 = ax i is (asymptotic) stable if a < 1. A LTI system x i+1 = Ax i is (asymptotically) stable if all eigenvalues of A is (strictly) inside the unit circle (length of eigenvalue less than one). 20/48

21 The Objective function N N 1 J = φ[x N ]+ i=0 L i (x i,u i ) R scalar function(s) consists of additive local terms (additve seperable terms) terminal contribution φ[x N ] running cost, stage cost, interval cost L i (x i,u i ) 21/48

22 Example: Max harvesting Consider a system described by the Schaefer model. Here we might be in a situation in which we want to maximize the total catch (over a longer peiod of time). First idea: For fixed N drive the system from x 0 along a trajectory such that N 1 J = u i i=0 is maximized. Pitch fall in optimization: Be careful in what you ask for - you might actually get it. 22/48

23 Example: Minimum consumption Consider a situation in which u i is related to consumption e.g. of some material or cost (price). For fixed N drive the system from x 0 to x N such that N 1 J = p i u i i=0 is minimized. p i is a price. E.g. consumption of electric energy. 23/48

24 Example: Minimum Energy/Power Often the square of the state or control variable is related to energy: E p = Ri 2 (current) E p = u2 R (voltage) E = 1 2 mv2 (moving mass) E = 1 2 kx2 (spring) Control system such that N 1 J = x 2 i i=0 is minimized. Often it is compromise between objectives. N 1 J = x 2 i +ρu2 i ρ 0 i=0 24/48

25 Quadratic cost For a multivariate system (x R n, u R m ): N 1 J = x T i Qx i +u T i Ru i R 0 Q 0 i=0 For example: x T Qx = [ x 1 x 2 ] [ ][ x1 x 2 ] = x x x 1x 2 The quadratic cost is often used in connection to minimizing the (square of) the distance (often to the origin - as in the example above). 25/48

26 Example: Minimum time Drive the system from x 0 to x N such that N 1 J = 1 = N i=0 is minimized. The length, N, of the period is then a part of the optimization. In this case or L = 1 φ = 0 L = 0 φ = N Often some constraints on the states or the decisions are involved. 26/48

27 Free Dynamic Optimization (D) Minimize J (ie. determine the sequence u i, i = 0,..., N 1) where: N 1 J = φ[x N ]+ subject to i=0 L i (x i,u i ) x i+1 = f i (x i,u i ) i = 0, 1,... N 1 x 0 = x 0 Here N, x 0 (and f, L and φ) are given. This is the Bolza formulation. The alternative is a Mayer formulation. 27/48

28 The Mayer form The problem can also be formulated as a minimization of: J = φ[ x N ] [ ] x x = z subject to x i+1 = f i ( x i,u i ) i = 0, 1,... N 1 x 0 = x 0 This is an equivalent formation of the Dynamic Optimization problem. The link from the Bolza to the Mayer formulation is given by: Minimize J = φ[ x N ] = φ(x N )+z N subject to: [ ] [ x = z i+1 f i (x i,u i ) z i +L i (x i,u i ) ] i = 0, 1,... N 1 [ x z ] 0 [ ] x0 = 0 28/48

29 Pause 29/48

30 Static Optimization with equality contraints - a recap Consider the problem of minimizing (find z R n ) L(z) R with respect to the (m) constraints g(z) = 0 R m Introduce the Lagrange multiplier, λ R m and define the Lagrange function: m J L (z,λ) = L(z)+λ T g(z) = L(z)+ λ k g k (z) k=1 The KKT (necessary) conditions are: λ J L = 0 g(z) = 0 z J L = 0 Stationarity Stated otherwise: (here z and λ). The Lagrangian should be stationary with respect to all variables Interpretation of Lagrange multipliers? 30/48

31 Euler-Lagranges equations N 1 J = φ[x N ]+ subject to i=0 L i (x i,u i ) x i+1 = f i (x i,u i ) i = 0, 1,... N 1 x 0 = x 0 Define a Lagrange multiplier vector, λ i+1, for each equality constraints x i+1 = f i (x i,u i ) i = 0, 1,... N 1 x 0 = x 0 and form the Lagrangian function: N 1 N 1 J L = φ[x N ]+ L i (x i,u i )+ λ T j+1 [f j(x j,u j ) x j+1 ]+λ T 0 [x 0 x 0 ] i=0 j=0 Necessarily condition: stationarity wrt. to x i, λ i and u i. 31/48

32 Stationarity wrt. λ N 1 N 1 J L = φ[x N ]+ L i (x i,u i )+ λ T j+1 [f j(x j,u j ) x j+1 ]+λ T 0 [x 0 x 0 ] i=0 j=0 Stationarity wrt. λ j+1 (i.e. wrt. λ 1... λ N ) gives f j (x j,u j ) x j+1 = 0 j = 0, 1,... N 1 or simply the state equation: x i+1 = f i (x i,u i ) j = 0, 1,... N 1 Stationarity wrt. λ 0 gives: x 0 = x 0 32/48

33 Stationarity wrt. x N 1 N 1 J L = φ[x N ]+ L i (x i,u i )+ λ T j+1 [f j(x j,u j ) x j+1 ]+λ T 0 [x 0 x 0] i=0 j=0 Stationarity wrt. x i (i = 1,... N 1) gives x L i(x i,u i )+λ T i+1 x f i(x i,u i ) λ T i = 0 Notice results for j = i and j +1 = i. Same result for i = 0. Result for i = N stated below. or the costate equation: λ T i = x L i(x i,u i )+λ T i+1 x f i(x i,u i ) i = 0, 1,... N 1 Stationarity wrt. x N gives λ T N = x N φ 33/48

34 Stationarity wrt. u Cut and paste - and use some colors: N 1 N 1 J L = φ[x N ]+ L i (x i,u i )+ λ T j+1 [f j(x j,u j ) x j+1 ]+λ T 0 [x 0 x 0 ] i=0 j=0 Stationarity wrt. u i gives the optimality condition or the stationarity condition: 0 = u L i(x i,u i )+λ T i+1 u f i(x i,u i ) 34/48

35 Euler-Lagrange equations Actually, the discrete Euler-Lagrange (EL) equations. Consider the (the Bolza formulation of the) problem of minimizing J, where N 1 J = φ[x N ]+ subject to i=0 L i (x i,u i ) x i+1 = f i (x i,u i ) x 0 = x 0 The EL equations (KKT conditions) are (for i = 0, 1,... N 1): x i+1 = f i (x i,u i ) λ T i = x L i(x i,u i )+λ T i+1 x f i(x i,u i ) 0 = u L i(x i,u i )+λ T i+1 u f i(x i,u i ) with boundary conditions x 0 = x 0 λ T N = φ x N This is a two-point boundary value problem (TPBVP) with N(2n + m) unknowns and equations. 35/48

36 Euler Born 15 April 1707 Basel, Switzerland. Died: 18 September 1783 (aged 76) Saint Petersburg, Russian Empire. Residence: Kingdom of Prussia, Russian Empire, Switzerland. Nationality: Swiss. Alma mater: University of Basel. Doctoral advisor: Johann Bernoulli. Doctoral students: Nicolas Fuss, Johann Hennert, Joseph Louis Lagrange. 36/48

37 Lagrange Born: Giuseppe Luigi Lagrancia, 25 January 1736 Turin, Piedmont-Sardinia. Died: 10 April 1813 (aged 77) Paris, France. Residence: Piedmont, France, Prussia. Citizenship: Kingdom of Sardinia, France. Nationality: Italian, French. Institutions: Ecole Polytechnique. Doctoral advisor: Leonhard Euler. Doctoral students: Joseph Fourier, Giovanni Plana, Simeon Poisson. 37/48

38 Example: Optimal stepping (D-time version of motion control) Consider the toy (or black board) problem of stepping the system x i+1 = x i +u i i = 0, 1,... N 1 from an original position x 0 towards origin in N steps. This is related to minimizing J 1 = 1 2 x2 N If the original objective and the step power has same weight (for the sake of simplicity) then the steps should be found as the minimizing strategy of J = 1 N 1 2 x2 N + i=0 1 2 u2 i In this case the EL equations are: x i+1 = x i +u i x 0 = x 0 λ i = λ i+1 λ N = x N 0 = u i +λ i+1 38/48

39 Example: Optimal stepping The last two are easily solved λ i = x N u i = x N which in the state equation give x i+1 = x i x N or x i = x 0 ix N For i = N this results in and x N = 1 N +1 x 0 u i = 1 N +1 x 0 1 x i = x 0 i N +1 x 0 λ i = 1 N +1 x 0 This problem has a degree of simplicity that allow us to find an analytical solution. 39/48

40 Types of solutions Examples: Analytical solutions (for very simple problems) Semi analytical solutions (for semi simple problems eg. the LQ problem) Numerical solutions 40/48

41 The Hamiltonian function Introduce the Hamiltonian function: H i (x i,u i,λ i+1 ) = L i (x i,u i )+λ T i+1 f i(x i,u i ) Then the EL can be written in a condensed form: x T i+1 = λ i+1 H i λ T i = x H i 0 = u H i x 0 = x 0 λ T N = x φ u H i is the gradient of J wrt. u i and is (also) called the pulse response sequence. λ T 0 is the gradient of J wrt. x 0. Notice the EL equations are necessary conditions. They are also sufficient if L i (x i,u i ) is a convex function (in x and u) f i (x i,u i ) is a linear function (in x and u). 41/48

42 Lesson learned (7) Minimize J (ie. determine the sequence u i, i = 0,..., N 1) where: N 1 J = φ[x N ]+ subject to i=0 L i (x i,u i ) x i+1 = f i (x i,u i ) i = 0, 1,... N 1 x 0 = x 0 Defining the Hamiltonian function H i = L i (x i,u i )+λ T i+1 f i(x i,u i ) The Euler-Lagrange equations can be written as: x i+1 = f i λ T i = x H 0 = u H x 0 = x 0 λ T N = x φ 42/48

43 Reading guidance Slides, Exercises, solutions and m-files on: Reading: DO: 5-11, (Ch. 2 except continuous time problems). 43/48

44 Notation Let s be a scalar, x a (column) vector x 1 x 2 x =. x n and f a vector function (of dim m). Then x s = ( x 1 s ie. a row vector. x 2 s... s ) x n Furthermore, the Jacobian is defined as: f x 1 f 1 x x f = f x 2 f 1 x f x m f 1 x m... 2 f x 1 n f x 2 n. x n f m 44/48

45 Notation The most common examples are: x Ax = A x yt x = y T x xt y = y T x xt Qx = 2x T Q 45/48

46 Rene Victor Valqui Vidal 46/48

47 Notation A comment on notation We use x i and f i (x i,u i ) rather than x(i) and f(i, x(i), u(i)) The curse of x vector containing the state variable first coordinate (in a position) decision variable in static optimization Hack(s): use s as symbol for the state use z as symbol for first coordinate use u as decision variable. 47/48

48 Other practicalities Campus net (cn.dtu.dk, now inside.dtu.dk). Course home page ( Gbar download (Matlab) ( Report errors and fishy elements in slides and lecture notes. suggestions for improvements. Include course id (42111) and your study id in s. 48/48