Optimization, Part 2 (november to december): mandatory for QEM-IMAEF, and for MMEF or MAEF who have chosen it as an optional course.

Transcription

1 Paris. Optimization, Part 2 (november to december): mandatory for QEM-IMAEF, and for MMEF or MAEF who have chosen it as an optional course. Philippe Bich (Paris 1 Panthéon-Sorbonne and PSE) Paris, 2016.

2 Chapter 1: duality theory. Introductory example The diet problem A student wants to prepair a tasty afternoon Two choice of food purchased from a bakery. Brownies (50 cents each) or mini cheese cakes (80 cents each). To simplify, let us say each cake is made of chocolate or sugar or cream-cheese. Moreover, the student wants to do a diet, and he has decided that he has minimal requirements for each ingredient (sugar,...) The following array summarizes everything: Chocolate Sugar CreamCheese UnitCost Brownie Cheesecake Requirements

3 Chapter 1: duality theory. Section 1: introduction Chocolate Sugar CreamCheese UnitCost Brownie Cheesecake Requirements If x 1 is the amount of brownies, x 2 the amount of cheescakes, the problem of the student is the following: under the constraints 3x 1 6, 2x 1 + 4x 2 10, 2x 1 + 5x 2 8, x 1 0, x 2 0. min 50x x 2

4 Chapter 1: duality theory. Section 1: introduction Now, imagine a supplier propose to sell directly the ingredients (chocolate,...) to the bakery. He wants to decide of the unit price of chocolate (y 1 ) of sugar (y 2 ) and of cream cheese (y 3 ) To maximize his revenue when he sells exactly what needs the bakery (6 unit of chocolate,...) Thus, the supplier maximizes 6y y 2 + 8y 3. The constraints are that the prices of each cake, given the price of ingredient, should be below what is ready to pay the student. That is, 3y 1 + 2y 2 + 2y 3 50 and 0y 1 + 4y 2 + 5y 3 80 and finally, y 1, y 2, y 3 0. Chocolate Sugar CreamCheese UnitCost Brownie Cheesecake Requirements

5 Chapter 1: duality theory. Section 1: introduction Definition: closed half-space H of R n : it can be written n H = {x = (x 1,..., x n ) R n : a i x i b} for some reals a i and b. Definition: convex polyhedron: It is a finite intersection of closed half-space of R n. Definition: a face of a convex set C if a convex subset F C such that if x, y, z are distinct points in C such that x = λy + (1 λ)z for some λ [0, 1] and x F, then y, z are in F. Definition: edges of C are faces that can be written [x, y] for some x, y in C (we say also 1 dimensional face). Definition: vertices of C are faces that can be written {x} for some x in C (we say also 0 dimensional face, or also extremal point). Remark: a convex polyhedron is closed, but may not be compact. i=1

6 Chapter 1: duality theory. Section 2: geometry of the problem Theorem: compact polyhedron Every compact and convex polyhedron C of R n can be written for some x 1,..., x n in R n. C = co{x 1,..., x n }

7 Chapter 1: duality theory. Section 2: geometry of the problem Let C be a convex polyhedron. Definition: A supporting hyperplan of C is an affine hyperplane H = {x = (x 1,..., x n ) R n : n i=1 a ix i = b} which intersects C on a face of C. Proposition: If H = {x = (x 1,..., x n ) R n : n i=1 a ix i = b} supporting hyperplan of C, then either: (1) For every x C, a.x b (in this case, we say that a = (a 1,..., a n ) points inward C ) (2) or for every x C, a.x b (in this case, we say that a = (a 1,..., a n ) points outward C )

8 Chapter 1: Duality theory. Section 2: geometry of the problem Theorem Consider the problem max l.x x=(x 1,...,x n) C where C is a convex polyhedron of R n and l = (l 1,..., l n ) R n. Then: (1) either this problem has no solution and the value of the problem is +, (2) or the set of a solution is a face F of C. In this last case, there exists b R such that H = {x = (x 1,..., x n ) R n : n i=1 l ix i = b} is a supporting hyperplan of C, and l = (l 1,..., l n ) points outward C. In particular, if C is a compact and convex polyhedron, that is C = co{x 1,..., x n }, then the set of solutions is nonempty and is a Face of C.

9 Chapter 1: duality theory. Section 2: geometry of the problem Example 1 Maximize x 1 + x 2 such that x 1 + 2x 2 4, 4x 1 + 2x 2 12, x 1 + x 2 1, x 1 0 and x2 0. Example 2 Maximize x 1 + x 2 such that x 1 + x 2 4, 4x 1 + 2x 2 12, x 1 + x 2 1, x 1 0 and x2 0.

10 Chapter 1: duality theory. Section 3: Terminology Definition: An optimization problem if unfeasible if the set of constraint is empty.

11 Chapter 1: duality theory. Section 4: duality theorems Theorem Let A is a m n matrix. Consider (P) we will call the primal problem, and (Q) max i=1,...,m, n j=1 a ij x j b i,x j 0 j=1 min j=1,...,n, m i=1 a ij y i c j,y i 0 i=1 n c j.x j m b i.y i we will call the dual problem. Weak duality property If x = ( x 1,.., x n) is a feasible point of (P), and ȳ = (ȳ 1,.., ȳ n) is a feasible point of (Q), then n j=1 c j. x j m i=1 b i.ȳ i. Strong duality property (admitted) If one optimization problem has a finite optimal solution, then so does has the other, and (Val(P) = Val(Q)). Unboundedness property If one optimization problem has an unbounded solution, then the other one is unfeasible.

12 Chapter 1. Dynamic programming. Finite horizon case You have a cake of length 1. Each day from today, you can consume c t. The more you consume one day, the happier your are this day. But you would like to have some cake tomorrow and the other days...thus, trade off. Mathematically...

13 Chapter 1. Dynamic programming. Finite horizon case you have to share 1 unit of good through time: let x t the share at time t, so that + i=0 x i = 1. Let C the set of sequences (x n ) such that this condition is true. Consider the following maximization problem + max β i u(x i ). (x n) C i=0 Here β ]0, 1[ is a discounted factor, and u a bounded and continuous utility function. Existence of a solution? Computation of the value?

14 Chapter 2: Dynamic programming: finite horizon problem Problem Maximize with respect to a 0 R,..., a T 1 R the function under the constraints. T 1 V (s, a) = ( β t f t(s t, a t)) + β T v T (s T ) t=0 t = 0,..., T 1, s t+1 = g t(s t, a t), s 0 given β discount factor, f t(s t, a t) payoff at t, v T (s T ) terminal payoff at T. g t describes the law of motion (how passing from s t to s t+1. ) s = (s 0, s 1,..., s T ) =state variable, describes the system. e.g. s t=size of the cake at t. a = (a 0, s 1,..., a T 1 )=control variable (or decision variable). e.g. a t= share of the cake you choose at date t.

15 Chapter 2: Dynamic programming: finite horizon problem To simplify problem, we discount the constraints: Problem Maximize with respect to a 0 R,..., a T 1 R the function under the constraints T 1 V (s, a) = ( β t f t(s t, a t)) + β T v T (s T ) t=0 t = 0,..., T 1, β t+1 (s t+1 g t(s t, a t)) = 0 Then we write the first order necessary condition, where λ 1,..., λ T multiplicators associated to constraints t = 0,..., T 1.

16 Chapter 2: Dynamic programming: finite horizon problem If f t, g t C 1, at solution, FOC gives: (2) Euler equation t = 0,..., T 1: (3) t = 1,..., T 1: f t a t (a t, s t) = λ t+1 β gt a t (a t, s t) f t s t (a t, s t) = λ t+1 β gt s t (a t, s t) + λ t (4) (5) t = 0,..., T 1 v s T (a t, s t) = λ T s t+1 = g t(s t, a t)

17 Chapter 2: Dynamic programming: finite horizon problem; INTERPRETATION (2) Euler equation t = 0,..., T 1: f t a t (a t, s t) = λ t+1 β gt a t (a t, s t) Interpretation: if one add 1 to a t (=share of the cake consumed at t=today). positive effect computed with payoff today approximately measured by f t a t (a t, s t). negative effect on utility tomorrow (because less cake!), expressed today, approximately measured by βλ t+1 g t a t (a t, s t) if λ t+1 interpreted as shadow price of cake tomorrow.

18 Chapter 2: Dynamic programming: finite horizon problem: INTERPRETATION (3) t = 1,..., T 1: f t s t (a t, s t) = λ t+1 β gt s t (a t, s t) + λ t Interpretation: if one add 1 to the size of cake s t today i.e. at t. Value of this additional unit can be measured through shadow price λ t of cake at t, which gives an increase of λ t. Value of this additional unit can be measured through additional payoff today plus discounted additional payoff tomorrow, that is f t s t (a t, s t) + λ t+1 β gt s t (a t, s t)

19 Chapter 2: Dynamic programming: finite horizon problem: INTERPRETATION (4) v s T (a T, s T ) = λ T Interpretation: if one add 1 to the size of the cake s T at the final date T Value of this additional unit can be measured through shadow price λ T of cake at T, which gives.value of this additional unit can be measured through additional payoff at T, that is λ T v s T (a T, s T )

20 Chapter 2: Dynamic programming: finite horizon problem: Example 1 Consider a consumer living for two periods t = 0, 1. He derives utility U : [0, + [ [0, + [, assumed C 1, from consuming a good c at each period. The initial endowment of the good is w. The consumer can borrow and lend intertemporally at an interest rate r. Suppose the intertemporal utility function V is separable and stationary, so that V (c 0, c 1 ) = U(c 0 ) + βu(c 1 ) where β stands for the discount rate. Find the optimal consumption across both periods. If one assume that U(.) is strictly concave, find a necessary and sufficient condition such that c 0 > c 1 at optimum.

21 Chapter 2: Dynamic programming: finite horizon problem Problem Maximize with respect to a 0 R,..., a T 1,... R the function + V (s, a) = ( β t f t (s t, a t )) under the constraints s 0 given. t=0 t = 0,..., T 1, s t+1 = g t (s t, a t ) To insure convergence, assume β ]0, 1[ (discount factor). Each f t is bounded.

22 Chapter 3: Dynamic programming: infinite horizon problem Problem Maximize with respect to a 0 R,..., a T 1,... R the function + V (s, a) = ( β t f t (s t, a t )) under the constraints s 0 given. t=0 t = 0,..., T 1, s t+1 = g t (s t, a t ) To solve it, method: write the FOC between two periods t and t + 1. This gives in general optimal solutions a. We shall see another method with Bellman principle.

23 Chapter 4: Dynamic programming: Bellman principle Recall the previous problem:

24 Chapter 4: Dynamic programming: Bellman principle Problem Maximize with respect to a 0 A,..., a T 1 A (where A set of actions) the function under the constraints T 1 V (s, a) = ( β t f t (s t, a t )) + β T v T (s T ) t=0 t = 0,..., T 1, s t+1 = g t (s t, a t ), s 0 given To avoid technical cases, assume each f t is bounded. Recall we maximize with respect to actions for which there exists some states satisfying the feasibility conditions above.

25 Chapter 4: Dynamic programming: Bellman principle Problem Maximize with respect to a 0 A,..., a T 1 A (where A set of actions) the function under the constraints T 1 V (s, a) = ( β t f t (s t, a t )) + β T v T (s T ) t=0 t = 0,..., T 1, s t+1 = g t (s t, a t ), s 0 given To avoid technical cases, assume each f t is bounded. Recall we maximize with respect to actions for which there exists some states satisfying the feasibility conditions above.

26 Chapter 4: Dynamic programming: Bellman principle Problem Maximize with respect to a 0 A,..., a T 1 A (where A set of actions) the function under the constraints T 1 V (s, a) = ( β t f t (s t, a t )) + β T v T (s T ) t=0 t = 0,..., T 1, s t+1 = g t (s t, a t ), s 0 given To avoid technical cases, assume each f t is bounded. Recall we maximize with respect to actions for which there exists some states satisfying the feasibility conditions above.

27 Chapter 4: Dynamic programming: Bellman principle For every state s S (set of states) and time t {0, 1,..., T 1}, let us define T 1 V t (s) = max( β k t f k (s k, a k )) + β T t v T (s T ) k=t Where the maximization is with respect to a t, a t+1,..., a T 1 for which there exists s t+1,..., s T such that k = t,..., T 1, s k+1 = g k (s k, a k ), s t = s given In particular, we are looking for V 0 (s 0 ). Idea: 1. V T 1 (s) easy to compute for every s S (by convention, if not feasible). 2. Then we can compute V T 2 (.) from V T 1 (s), and V T 3 (.) from V T 2 (s), etc etc... by induction, thanks to the following Bellman principle:

28 Chapter 4: Dynamic programming: Bellman principle For every state s S (set of states) and time t {0, 1,..., T 1}, let us define T 1 V t (s) = max( β k t f k (s k, a k )) + β T t v T (s T ) k=t Where the maximization is with respect to a t, a t+1,..., a T 1 for which there exists s t+1,..., s T such that k = t,..., T 1, s k+1 = g k (s k, a k ), s t = s given In particular, we are looking for V 0 (s 0 ). Idea: 1. V T 1 (s) easy to compute for every s S (by convention, if not feasible). 2. Then we can compute V T 2 (.) from V T 1 (s), and V T 3 (.) from V T 2 (s), etc etc... by induction, thanks to the following Bellman principle:

29 Chapter 4: Dynamic programming: Bellman principle For every state s S (set of states) and time t {0, 1,..., T 1}, let us define T 1 V t (s) = max( β k t f k (s k, a k )) + β T t v T (s T ) k=t Where the maximization is with respect to a t, a t+1,..., a T 1 for which there exists s t+1,..., s T such that k = t,..., T 1, s k+1 = g k (s k, a k ), s t = s given In particular, we are looking for V 0 (s 0 ). Idea: 1. V T 1 (s) easy to compute for every s S (by convention, if not feasible). 2. Then we can compute V T 2 (.) from V T 1 (s), and V T 3 (.) from V T 2 (s), etc etc... by induction, thanks to the following Bellman principle:

30 Chapter 4: Dynamic programming: Bellman principle For every state s S (set of states) and time t {0, 1,..., T 1}, let us define T 1 V t (s) = max( β k t f k (s k, a k )) + β T t v T (s T ) k=t Where the maximization is with respect to a t, a t+1,..., a T 1 for which there exists s t+1,..., s T such that k = t,..., T 1, s k+1 = g k (s k, a k ), s t = s given In particular, we are looking for V 0 (s 0 ). Idea: 1. V T 1 (s) easy to compute for every s S (by convention, if not feasible). 2. Then we can compute V T 2 (.) from V T 1 (s), and V T 3 (.) from V T 2 (s), etc etc... by induction, thanks to the following Bellman principle:

31 Chapter 4: Dynamic programming: Bellman principle For every state s S (set of states) and time t {0, 1,..., T 1}, let us define T 1 V t (s) = max( β k t f k (s k, a k )) + β T t v T (s T ) k=t Where the maximization is with respect to a t, a t+1,..., a T 1 for which there exists s t+1,..., s T such that k = t,..., T 1, s k+1 = g k (s k, a k ), s t = s given In particular, we are looking for V 0 (s 0 ). Idea: 1. V T 1 (s) easy to compute for every s S (by convention, if not feasible). 2. Then we can compute V T 2 (.) from V T 1 (s), and V T 3 (.) from V T 2 (s), etc etc... by induction, thanks to the following Bellman principle:

32 Chapter 4: Dynamic programming: Bellman principle For every state s S (set of states) and time t {0, 1,..., T 1}, let us define T 1 V t (s) = max( β k t f k (s k, a k )) + β T t v T (s T ) k=t Where the maximization is with respect to a t, a t+1,..., a T 1 for which there exists s t+1,..., s T such that k = t,..., T 1, s k+1 = g k (s k, a k ), s t = s given In particular, we are looking for V 0 (s 0 ). Idea: 1. V T 1 (s) easy to compute for every s S (by convention, if not feasible). 2. Then we can compute V T 2 (.) from V T 1 (s), and V T 3 (.) from V T 2 (s), etc etc... by induction, thanks to the following Bellman principle:

33 Chapter 4: Dynamic programming: Bellman principle Bellman Principle For every time t {1, 2,..., T 1}, and every state s S (at time t 1), one has: V t 1 (s) = max a t 1 A {f t 1(s, a t 1 )) + βv t (g t 1 (s, a t 1 )} Remark also that Proof. V T 1 (s) = max a T 1 A {f T 1(s, a T 1 )) + βv T (g T 1 (s, a T 1 )}

34 Chapter 4: Dynamic programming: Bellman principle Bellman Principle For every time t {1, 2,..., T 1}, and every state s S (at time t 1), one has: V t 1 (s) = max a t 1 A {f t 1(s, a t 1 )) + βv t (g t 1 (s, a t 1 )} Remark also that Proof. V T 1 (s) = max a T 1 A {f T 1(s, a T 1 )) + βv T (g T 1 (s, a T 1 )}

35 Chapter 4: Dynamic programming: Bellman principle Bellman Principle For every time t {1, 2,..., T 1}, and every state s S (at time t 1), one has: V t 1 (s) = max a t 1 A {f t 1(s, a t 1 )) + βv t (g t 1 (s, a t 1 )} Remark also that Proof. V T 1 (s) = max a T 1 A {f T 1(s, a T 1 )) + βv T (g T 1 (s, a T 1 )}

36 Chapter 4: Dynamic programming: Bellman principle Bellman Principle For every time t {1, 2,..., T 1}, and every state s S (at time t 1), one has: V t 1 (s) = max a t 1 A {f t 1(s, a t 1 )) + βv t (g t 1 (s, a t 1 )} Remark also that Proof. V T 1 (s) = max a T 1 A {f T 1(s, a T 1 )) + βv T (g T 1 (s, a T 1 )}

37 Chapter 4: Dynamic programming: Bellman principle Consider max a 0,a 1,a 2,a 3 3 (10s t (0, 1)at 2 ) t=0 under the constraint s t+1 = s t + a t, s 0 = 0, a t 0. Here, a t investment at t, s t capital stock at t. We use Bellman optimal principle method : we begin to solve the problem till the end, and we do backward induction. At t = 3: given s 3, we solve max a 3 0 (10s 3 (0, 1)a 2 3 ). gives a 3 = 0 and v 3(s 3 ) = 10s 3 optimal value at t = 3 given s 3.

38 Chapter 4: Dynamic programming: Bellman principle Consider max a 0,a 1,a 2,a 3 3 (10s t (0, 1)at 2 ) t=0 under the constraint s t+1 = s t + a t, s 0 = 0, a t 0. Here, a t investment at t, s t capital stock at t. We use Bellman optimal principle method : we begin to solve the problem till the end, and we do backward induction. At t = 3: given s 3, we solve max a 3 0 (10s 3 (0, 1)a 2 3 ). gives a 3 = 0 and v 3(s 3 ) = 10s 3 optimal value at t = 3 given s 3.

39 Chapter 4: Dynamic programming: Bellman principle Consider max a 0,a 1,a 2,a 3 3 (10s t (0, 1)at 2 ) t=0 under the constraint s t+1 = s t + a t, s 0 = 0, a t 0. Here, a t investment at t, s t capital stock at t. We use Bellman optimal principle method : we begin to solve the problem till the end, and we do backward induction. At t = 3: given s 3, we solve max a 3 0 (10s 3 (0, 1)a 2 3 ). gives a 3 = 0 and v 3(s 3 ) = 10s 3 optimal value at t = 3 given s 3.

40 Chapter 4: Dynamic programming: Bellman principle At t = 2: given s 2, we solve max a s 2 (0, 1)a 2 2 +v 3(s 3 ) = 10s 2 (0, 1)a (s 2 +a 2 ). gives a2 = 50 and v 2 (s 2 ) = 10.s 2 (0, 1) (s ) = 20s optimal value at t = 2 given s 2. At t = 1: given s 1, we solve max 10s 1 (0, 1)a1 2 +v 2(s 2 ) = 10s 1 (0, 1)a (s 1+a 1 )+250 a 1 0 gives a 1 = 100 and v 1(s 1 ) = 30s optimal value at t = 1 given s 1. At t = 0: given s 1, we solve max 10s 0 (0, 1)a0 2 +v 1(s 1 ) = 10s 0 (0, 1)a s a 0 0 gives a 0 = 150 and v 1(s 0 ) = 3500.

41 Chapter 5: Dynamic programming: Section 1:Bellman method with infinite horizon and stationary assumptions Problem Assume f is a bounded function (to simplify), and β ]0, 1[. For an initial state s 0 S (set of states), denote V (s 0 ) the value of the following problem, where the variable is some real sequence a = (a t ) t 0 in A (set of actions): V (s 0 ) = under the constraints + sup β t f (s t, a t ) (a k ) k 0,(s k ) k 0 t=0 t 0, s t+1 = g(s t, a t ), s 0 given

42 Chapter 5: Dynamic programming: Section 1:Bellman method with infinite horizon and stationary assumptions infinite horizon Bellman Principle For every state s S, one has Proof. V (s) = sup{f (s, a)) + βv (g(s, a))} a A

43 Chapter 5: Dynamic programming: Section 2 : the Fixed-point method The fixed point method For every function V (.) from S to R, define B(V )(s) = max{f (s, a)) + βv (g(s, a))}. a By infinite horizon Bellman principle, B(V ) = V. Thus, we now raise the following problem: method to prove and construct some operator B (i.e. a function that associates to some function another function!) admits a fixed point.

44 Chapter 5: Dynamic programming: Section 2 : the Fixed-point method The fixed point method For every function V (.) from S to R, define B(V )(s) = max{f (s, a)) + βv (g(s, a))}. a By infinite horizon Bellman principle, B(V ) = V. Thus, we now raise the following problem: method to prove and construct some operator B (i.e. a function that associates to some function another function!) admits a fixed point.

45 Chapter 5: Dynamic programming: Section 2 : the Fixed-point method The fixed point method For every function V (.) from S to R, define B(V )(s) = max{f (s, a)) + βv (g(s, a))}. a By infinite horizon Bellman principle, B(V ) = V. Thus, we now raise the following problem: method to prove and construct some operator B (i.e. a function that associates to some function another function!) admits a fixed point.

46 Chapter 5: Dynamic programming: Section 2 : the Fixed-point method The fixed point method For every function V (.) from S to R, define B(V )(s) = max{f (s, a)) + βv (g(s, a))}. a By infinite horizon Bellman principle, B(V ) = V. Thus, we now raise the following problem: method to prove and construct some operator B (i.e. a function that associates to some function another function!) admits a fixed point.

47 Chapter 5: Dynamic programming: Section 2 : the Fixed-point method The fixed point method For every function V (.) from S to R, define B(V )(s) = max{f (s, a)) + βv (g(s, a))}. a By infinite horizon Bellman principle, B(V ) = V. Thus, we now raise the following problem: method to prove and construct some operator B (i.e. a function that associates to some function another function!) admits a fixed point.

48 Chapter 5: Dynamic programming: Section 2 : the Fixed-point method The fixed point method For every function V (.) from S to R, define B(V )(s) = max{f (s, a)) + βv (g(s, a))}. a By infinite horizon Bellman principle, B(V ) = V. Thus, we now raise the following problem: method to prove and construct some operator B (i.e. a function that associates to some function another function!) admits a fixed point.

49 Chapter 5: Dynamic programming: Section 3 : the Banach-Picard fixed-point theorem Norme. sur E, un IR-e.v.:. application de E dans IR + ; pour tout x E, x = 0 si et seulement si x = 0; pour tout x E, pour tout t IR, tx = t x ; pour tout (x, y) E E, x + y x + y.

50 Chapter 5: Dynamic programming: Section 3 : the Banach-Picard fixed-point theorem Norme. sur E, un IR-e.v.:. application de E dans IR + ; pour tout x E, x = 0 si et seulement si x = 0; pour tout x E, pour tout t IR, tx = t x ; pour tout (x, y) E E, x + y x + y.

51 Chapter 5: Dynamic programming: Section 3 : the Banach-Picard fixed-point theorem Norme. sur E, un IR-e.v.:. application de E dans IR + ; pour tout x E, x = 0 si et seulement si x = 0; pour tout x E, pour tout t IR, tx = t x ; pour tout (x, y) E E, x + y x + y.

52 Chapter 5: Dynamic programming: Section 3 : the Banach-Picard fixed-point theorem Norme. sur E, un IR-e.v.:. application de E dans IR + ; pour tout x E, x = 0 si et seulement si x = 0; pour tout x E, pour tout t IR, tx = t x ; pour tout (x, y) E E, x + y x + y.

53 Chapter 5: Dynamic programming: Section 3 : the Banach-Picard fixed-point theorem Norme. sur E, un IR-e.v.:. application de E dans IR + ; pour tout x E, x = 0 si et seulement si x = 0; pour tout x E, pour tout t IR, tx = t x ; pour tout (x, y) E E, x + y x + y.

54 Chapter 5: Dynamic programming: Section 3 : the Banach-Picard fixed-point theorem The space E endowed with a Norm. is said to be a Banach space if it is complete, i.e. every Cauchy sequence converges, that is... Example 1: every finite dimensional vector space endowed with any norm is a Banach space. Example 2: norm l p on l p, p = 1,..., +. Example 3: norm on the set of continuous functions from some compact set K to R. Proof for Example 3 (Important).

55 Chapter 5: Dynamic programming: Section 3 : the Banach-Picard fixed-point theorem The space E endowed with a Norm. is said to be a Banach space if it is complete, i.e. every Cauchy sequence converges, that is... Example 1: every finite dimensional vector space endowed with any norm is a Banach space. Example 2: norm l p on l p, p = 1,..., +. Example 3: norm on the set of continuous functions from some compact set K to R. Proof for Example 3 (Important).

56 Chapter 5: Dynamic programming: Section 3 : the Banach-Picard fixed-point theorem The space E endowed with a Norm. is said to be a Banach space if it is complete, i.e. every Cauchy sequence converges, that is... Example 1: every finite dimensional vector space endowed with any norm is a Banach space. Example 2: norm l p on l p, p = 1,..., +. Example 3: norm on the set of continuous functions from some compact set K to R. Proof for Example 3 (Important).

57 Chapter 5: Dynamic programming: Section 3 : the Banach-Picard fixed-point theorem The space E endowed with a Norm. is said to be a Banach space if it is complete, i.e. every Cauchy sequence converges, that is... Example 1: every finite dimensional vector space endowed with any norm is a Banach space. Example 2: norm l p on l p, p = 1,..., +. Example 3: norm on the set of continuous functions from some compact set K to R. Proof for Example 3 (Important).

58 Chapter 5: Dynamic programming: Section 3 : the Banach-Picard fixed-point theorem Definition A function f : E E is said to be contracting if there exists k ]0, 1[ such that (x, y) E E : f (x) f (y) k x y. Theorem (Banach-Picard) Let (E,. ) be a Banach space and f : E E a contracting function. Then: (i) there exists some fixed point x E of f, that is f ( x) = x. (ii) For every x 0 E fixed, the sequence defined inductively by x n+1 = f (x n ) converges to x. Proof.

59 Chapter 5: Dynamic programming: Section 3 : the Banach-Picard fixed-point theorem Definition A function f : E E is said to be contracting if there exists k ]0, 1[ such that (x, y) E E : f (x) f (y) k x y. Theorem (Banach-Picard) Let (E,. ) be a Banach space and f : E E a contracting function. Then: (i) there exists some fixed point x E of f, that is f ( x) = x. (ii) For every x 0 E fixed, the sequence defined inductively by x n+1 = f (x n ) converges to x. Proof.

60 Chapter 5: Dynamic programming: Section 3 : the Banach-Picard fixed-point theorem Definition A function f : E E is said to be contracting if there exists k ]0, 1[ such that (x, y) E E : f (x) f (y) k x y. Theorem (Banach-Picard) Let (E,. ) be a Banach space and f : E E a contracting function. Then: (i) there exists some fixed point x E of f, that is f ( x) = x. (ii) For every x 0 E fixed, the sequence defined inductively by x n+1 = f (x n ) converges to x. Proof.

61 Chapter 5: Dynamic programming: Section 3 : the Banach-Picard fixed-point theorem Definition A function f : E E is said to be contracting if there exists k ]0, 1[ such that (x, y) E E : f (x) f (y) k x y. Theorem (Banach-Picard) Let (E,. ) be a Banach space and f : E E a contracting function. Then: (i) there exists some fixed point x E of f, that is f ( x) = x. (ii) For every x 0 E fixed, the sequence defined inductively by x n+1 = f (x n ) converges to x. Proof.

62 Chapter 5: Dynamic programming: Section 3 : the Banach-Picard fixed-point theorem Definition A function f : E E is said to be contracting if there exists k ]0, 1[ such that (x, y) E E : f (x) f (y) k x y. Theorem (Banach-Picard) Let (E,. ) be a Banach space and f : E E a contracting function. Then: (i) there exists some fixed point x E of f, that is f ( x) = x. (ii) For every x 0 E fixed, the sequence defined inductively by x n+1 = f (x n ) converges to x. Proof.

63 Chapter 5: Dynamic programming: Section 3 : the Banach-Picard fixed-point theorem Theorem The Banach-Picard theorem is true if we only assume that some iterate f k is contracting for some k 1. Proof.

64 Chapter 5: Dynamic programming: Section 3 : the Banach-Picard fixed-point theorem Theorem The Banach-Picard theorem is true if we only assume that some iterate f k is contracting for some k 1. Proof.

65 Chapter 5: Dynamic programming: Section 4: Blackwell theorem Theorem Let B(X), endowed with., the set of bounded functions from some nonempty subset X. Let L some closed (for the norm) vector subspace of B(X) containing every constant functions. Let T : L L such that: i) T is increasing, in the sense: f g implies T (f ) T (g). ii) There exists β ]0, 1[ such that for every constant function c, T (f + c) T (f ) + β.c for every f L. Then T admits a fixed point. Here f g means f (x) g(x) for every x. The same for T (f ) T (g).

66 Chapter 5: Dynamic programming: Section 4: Blackwell theorem Theorem Let B(X), endowed with., the set of bounded functions from some nonempty subset X. Let L some closed (for the norm) vector subspace of B(X) containing every constant functions. Let T : L L such that: i) T is increasing, in the sense: f g implies T (f ) T (g). ii) There exists β ]0, 1[ such that for every constant function c, T (f + c) T (f ) + β.c for every f L. Then T admits a fixed point. Here f g means f (x) g(x) for every x. The same for T (f ) T (g).

67 Chapter 5: Dynamic programming: Section 4: Blackwell theorem Theorem Let B(X), endowed with., the set of bounded functions from some nonempty subset X. Let L some closed (for the norm) vector subspace of B(X) containing every constant functions. Let T : L L such that: i) T is increasing, in the sense: f g implies T (f ) T (g). ii) There exists β ]0, 1[ such that for every constant function c, T (f + c) T (f ) + β.c for every f L. Then T admits a fixed point. Here f g means f (x) g(x) for every x. The same for T (f ) T (g).

68 Chapter 5: Dynamic programming: Section 4: Blackwell theorem Theorem Let B(X), endowed with., the set of bounded functions from some nonempty subset X. Let L some closed (for the norm) vector subspace of B(X) containing every constant functions. Let T : L L such that: i) T is increasing, in the sense: f g implies T (f ) T (g). ii) There exists β ]0, 1[ such that for every constant function c, T (f + c) T (f ) + β.c for every f L. Then T admits a fixed point. Here f g means f (x) g(x) for every x. The same for T (f ) T (g).

69 Chapter 5: Dynamic programming: Section 4: Blackwell theorem Theorem Let B(X), endowed with., the set of bounded functions from some nonempty subset X. Let L some closed (for the norm) vector subspace of B(X) containing every constant functions. Let T : L L such that: i) T is increasing, in the sense: f g implies T (f ) T (g). ii) There exists β ]0, 1[ such that for every constant function c, T (f + c) T (f ) + β.c for every f L. Then T admits a fixed point. Here f g means f (x) g(x) for every x. The same for T (f ) T (g).

70 Chapter 5: Dynamic programming: Section 4: Blackwell theorem Theorem Let B(X), endowed with., the set of bounded functions from some nonempty subset X. Let L some closed (for the norm) vector subspace of B(X) containing every constant functions. Let T : L L such that: i) T is increasing, in the sense: f g implies T (f ) T (g). ii) There exists β ]0, 1[ such that for every constant function c, T (f + c) T (f ) + β.c for every f L. Then T admits a fixed point. Here f g means f (x) g(x) for every x. The same for T (f ) T (g).

71 Chapter 5: Dynamic programming: Section 4: Blackwell theorem Theorem Let B(X), endowed with., the set of bounded functions from some nonempty subset X. Let L some closed (for the norm) vector subspace of B(X) containing every constant functions. Let T : L L such that: i) T is increasing, in the sense: f g implies T (f ) T (g). ii) There exists β ]0, 1[ such that for every constant function c, T (f + c) T (f ) + β.c for every f L. Then T admits a fixed point. Here f g means f (x) g(x) for every x. The same for T (f ) T (g).

72 Chapter 5: Dynamic programming: Section 6: Existence of a solution of Bellman equation The equation B(V )(s) = max{f (s, a)) + βv (g(s, a)) is called Bellman equation. a Theorem (Existence) Assume S (state space) and A (action space) are compact, that f and g are continuous. Let L be the space of continuous functions from S to R, and T = B the Bellman operator. Then T satisfies the assumptions of Blackwell principle. In particular, there exists V L such that B(V ) = V.

73 Chapter 5: Dynamic programming: Section 6: Existence of a solution of Bellman equation The equation B(V )(s) = max{f (s, a)) + βv (g(s, a)) is called Bellman equation. a Theorem (Existence) Assume S (state space) and A (action space) are compact, that f and g are continuous. Let L be the space of continuous functions from S to R, and T = B the Bellman operator. Then T satisfies the assumptions of Blackwell principle. In particular, there exists V L such that B(V ) = V.

74 Chapter 5: Dynamic programming: Section 6: Existence of a solution of Bellman equation What can we do from a solution V of bellman equation? Let V a solution of B(V ) = V. Do we have for every state s 0 S, + V (s 0 ) = sup β t f (s t, a t ) a,s t=0 under the constraints t 0, s t+1 = g(s t, a t ), s 0 given Let assume f continuous and S and A compact.

75 Chapter 5: Dynamic programming: Section 6: Existence of a solution of Bellman equation What can we do from a solution V of bellman equation? Let V a solution of B(V ) = V. Do we have for every state s 0 S, + V (s 0 ) = sup β t f (s t, a t ) a,s t=0 under the constraints t 0, s t+1 = g(s t, a t ), s 0 given Let assume f continuous and S and A compact.