Salient features of dynamic optimization problems Basics of dynamic programming The principle of optimality

Transcription

1

2

3 A Z D I A 4 2 B C E F G H J K 3 2 Z

4 D I A 4 2 B C E F G H J K 3 2 Z s x {, 2,..., n} r(x, s) x

5 s r(, A) = 2 r(, D) = 2 r(2, A) = 4 r(2, D) = 8 g(x, s) s s x g(, A) = B g(2, A) = C g(, D) = I g(2, D) = J x = s r(s, s ) r(a, B) = 2 r(g, I) = 2 r(a, C) = 4 r(g, J) = 8 g(s, s ) g(a, B) = B g(a, C) = C g(g, I) = I g(g, J) = J t s t s(t) c(abdgiz) = = 7 = c() c(abehkz) = = 8 = c(22) 4 r(s t, x t ) s t+ = g(x t, s t ), t = 0,..., 4 V t (s t ) t s t V t (s t ) = x 0,...,x 4 4 r(s t, x t ) s t+ = g(x t, s t ), t = 0,..., 4

6 h t (s t ) s t t h t (s t ) = x 0,...,x 4 4 r(s t, x t ) s t+ = g(x t, s t ), t = 0,..., 4 V t (s t ) = x t {r(x t, s t ) + V t+ (s t+ )} s t+ = g(x t, s t ) V t (s t ) = x t {r(x t, s t ) + V t+ (g(x t, s t ))} V t (s t ) = r [h t (s t ), s t ] + V t+ [g (h t (s t ), s t )] V t (s t ) = x t {r(x t, s t ) + V t+ (g(x t, s t ))} Z 0 A 4 2 B C D E F G H I J K 3 2 Z

7 V 5 (Z) = 0 h 5 (Z) = Z V 4 (K) = {2 + V 5 (Z)} = 2 h 4 (K) = Z V 4 (J) = { + V 5 (Z)} = h 4 (J) = Z V 4 (I) = {3 + V 5 (Z)} = 3 h 4 (I) = Z V 3 (H) = {4 + V 4 (J), 6 + V 4 (K)} = 5 h 3 (H) = J V 3 (G) = {2 + V 4 (I), 8 + V 4 (J)} = 5 h 3 (G) = I V 2 (F ) = {5 + V 3 (H)} = 0 h 2 (F ) = H V 2 (E) = {3 + V 3 (H)} = 8 h 2 (E) = H V 2 (D) = {3 + V 3 (G), 4 + V 3 (H)} = 8 h 2 (D) = G V (C) = {2 + V 2 (E), 6 + V (F )} = 0 h (C) = E V (B) = {7 + V 2 (D), 5 + V 2 (E)} = 3 h (B) = E V 0 (A) = {2 + V (B), 4 + V (C)} = 4 h 0 (A) = C β (0, ) {u t } β t r(x t, u t ) x t+ = g(x t, u t ) x 0 R r(x t, u t ) {(x t+, x t ) : x t+ g(x t, u t ), u t R} h x t u t {u s } s=0 u t = h(x t ) x t+ = g(x t, u t )

8 x 0 t = 0 h V (x) x X V (x 0 ) = {u s} s=0 β t r(x t, u t ) x t+ = g(x t, u t ) x 0 V (x 0 ) h x X {r(x, u) + βv ( x)}, x = g(x, u) u V (x) h x V (x), h(x) V (x) = {r(x, u) + βv [g(x, u)]} u h(x) V (x) = r[x, h(x)] + βv {g[x, h(x)]} V (x), h(x) r g j V j+ (x) = {r(x, u) + βv j( x)}, x = g(x, u), x u V 0 u t = h(x t ) h V f x y t [0, ] t (0, ) x y f(( t)x + ty) ( t)f(x) + tf(y) f(( t)x + ty) ( t)f(x) + tf(y)

9 f : R R z x y (z, f(z)) f f(y) f(x) ( t)f(x) + tf(y) (x, f(x)) (y, f(y)) x y x f f(x ) = x f(f(... f(x )... )) = x f y = x x x x 0 f : X X X δ 0 δ < d(f(x), f(y)) δd(x, y)

10 f f(t) f(x) f(y) x y x y X x y t f X x f(x ) = x x 0 X x i+ = f(x i ) x i x x 0 x d(f(x 0 ), f(x )) δd(x 0, x ) d(x, x ) δd(x 0, x ) d(x k, x ) δ k d(x 0, x ) 0 k f(x) = + 0.5x x R x = 2 f(x ) = f(2) = + 0.5(2) = 2 = x x = + 0.5x f (x) = 0.5 < f x 0

11 x j+ = f(x j ) j x j = x x 0 = 6 x = f(x 0 ) = = 4 x 2 = f(x ) = = 3 x 3 = f(x 2 ) = = 2.5 x 4 = f(x 3 ) = = 2.25 f f(t) x = 2 x 0 x r u (x, u) + βv {g(x, u)}g u (x, u) = 0 V (x) = r x [x, h(x)] + r u [x, h(x)]h (x) + βv {g[x, h(x)]} {g x [x, h(x)] + g u [x, h(x)]h (x)} u x = g(u) V (x) = r x [x, h(x)] x = g(u) r u (x t, u t ) + βr x (x t+, u t+ )g (u t ) = 0

12 u t x t+ u t x t V V V

13 V 0 = 0 V j V j+ (x) = {r(x, u) + βv j[g(x, u)]} u u = h 0 (x) V hj (x) = β t r[x t, h j (x t )] x t+ = g[x t, h j (x t )] j = 0 u = h j+ (x) j {r(x, u) + βv h j [g(x, u)]} u T u(c t ) = c t β (0, ) U(c 0, c,..., c T ) = T β t c t c t U A 0 r R + r A t+ = R(A t c t )

14 {c t } T A 0 T + V V 0 (A 0 ) = {c,a} T β t c t V 0 (A 0 ) = {c t} T β t c t A t+ = R(A t c t ) A t+ = R(A t c t ) t = 0,..., T A T + 0 t = T t = T t = T 2 T = t = T V T (A T ) = c T,A T + { c T } A T + = R(A T c T ), A T + 0 c T A T + c T = A T A T + R [ A T A ] T + A T + R A T + 0 A T + A T + A T + = 0 c T = A T V T (A T ) = A T t = T V T (A T ) = { c T + + β c T } A T = R (A T c T ), A T + = R(A T c T ), A T + 0 c T, c T, A T A T +

15 T c T A T c T A T + t = T c T = A T A T + = 0 V T (A T ) = { c T + β c T } c T :T,A T :T + { } = c T + β [ c T ] c T,A T c T,A T + = c T,A T { c T + βv T (A T )} A T = R(A T c T ) c T = A T A T R { A T [ A T A ] T + βv T (A T ) R c T R + βv T (A T ) = 0 = Rβc T V T (A T ) } V T (A) = A t = T V T (A T ) = A T = Rβc T A T A T = Rβc T Rβc T = R(A T c T ) c T = +β A T A T = Rβ +β A T V T (A T ) = c T + βv T (A T ) θ T t = T 2 = c T + β A T = c T + β [Rβc T ] = ( + β) c T + β β + β R = ( + β) A T ( + β) ( + β) β β + β R = ( + β) A T + θ T

16 V T 2 (A T 2 ) = { c T 2 + β c T + β 2 c T } A T = R(A T 2 c T 2 ), A T = R(A T c T ), A T + = R(A T c T ), A T + 0 c T 2 A T c T, c T, A T, A T + { V T 2 (A T 2 ) = c T 2:T, ct 2 + β c T + β 2 } c T A T :T + = c T 2,A T c T 2 + β c T :T, = c T 2,A T { c T 2 + βv T (A T )} [ c T + β c T ] A T :T + c T 2 = A T 2 A T R { [ A T 2 A ] } T + βv T (A T ) A T R c T 2 R + βv T (A T ) = 0 = Rβc T 2 V T (A T ) V T (A) = (+β) A+θ T t = T V T (A T ) = +β A T = Rβc T 2 +β A T A T = R(β + β 2 )c T 2 ( + β)rβc T 2 = R(A T 2 c T 2 ) c T 2 = +β+β 2 A T 2 A T = R(β+β2 ) +β+β 2 A T 2 V T 2 (A T 2 ) = c T 2 + βv T (A T ) = c T 2 + β[( + β) (A T ) + θ T ] = c T 2 + (β + β 2 ) [R(β + β 2 )c T 2] + βθ T = ( + β + β2 ) c T 2 + (β + β 2 )[ R (β + β 2 )] + βθ T = ( + β + β 2 ) A T 2 + θ T 2 θ T 2 = (β + 2β 2 ) R + (β + 2β 2 ) β ( + β + β 2 ) ( + β + β 2 )

17 t = T K t = T K V T K (A T K ) = { c T K + β c T K+ + + β K c T } A t+ = R(A t c t ), t = T K, T K +,..., T A T + 0 c T K A T K+ c T K, A T K+ c T K + β c T K = A T K A T K+ R V T K (A T K ) = [ c c T K+:T, T K+ + + β K c T ] A T K+2:T + = c T K, A T K+ { c T K + βv T K+ (A T K+ )} { [ A T K A T K+ A T K R ] + βv T K+ (A T K+ ) c T K R + βv T K+(A T K+ ) = 0 = Rβc T K V T K+(A T K+ ) } V T K+ (A) t c t A t+ T A T 0A T T +β A T Rβ +β A T T 2 +β+β A 2 T 2 Rβ +β +β+β A 2 T 2 K T K +β+ +β K A T K Rβ +β+ +βk A +β+ +β K T K = β β K+ A T K Rβ βk β K+ A T K

18 A T K+ = Rβ βk β K+ A T K K = T, T A = Rβ βt β T + A 0 βt A 2 = Rβ β T A 2 βt = (Rβ) β T + A 0 + t t βt A t = (Rβ) β T + A 0 c T K = β A β K+ T K t = T K c β t = β T + t A t = β β T + t = (Rβ) t β β T + A 0 ϕ + t t βt [(Rβ) β T + A 0 ] c t = t (Rβ) + ϕ 0 V 0 (A 0 ) T β t c t = T β t (t (Rβ) + ϕ) T T = (Rβ) β t t + ϕ = β β = β β + ( β T β T βt ( β T β T βt + βt β β t ) + βt (Rβ) + β ) (Rβ) +... β β T + + βt + A 0 β ϕ

19 + t t βt A t = (Rβ) β T + A 0 c t = (Rβ) t β β T + A 0 V 0(A 0) = β ( ) β T + β β T βt β βt (Rβ) + β β + βt + A T + 0 β T A t = (Rβ) t A 0 c t = (Rβ) t ( β)a 0 = ( β)a t V 0 (A 0 ) = β A β R + β β + ( β) ( β) 0 + ( β) 2 c t = ( β)a t A t+ = RβA t β β R t T T T 2 V t (A) A ( + β) A + θ T ( + β + β 2 ) A + θ T 2 K V T K (A T K ) = { c c T K, T K + βv T K+ (A T K+ )} A T K+ T K T K + V (A T K ) = { c c T K, T K + βv (A T K+ )} A T K+ V (A) = { c + βv c,a (A )}

20 { ) } V (A) = (A A + βv (A ) A R = RβcV (A ) V V (A) = (A A ) + βv (A ) R V (A) = c V (A) = c V (A ) = c = Rβ c c = Rβ u (c ) u (c) u (c) βu (c ) R V (A) = c,a { c + βv (A )} A = R(A c) V j+ (A) = c,a { c + βv j(a )} A = R(A c) V 0 (A) = 0 j V j (A) 0 0 A 2 ( + β) A + θ 2 3 ( + β + β 2 ) A + θ 3 A + β + β 2 + = β θ j V (A) = A + θ β

21 V (A) = β A + θ = RβcV (A ) A = R(A c) c = ( β)a A = RβA V (A) c + βv (A ) β A + θ [ ] A = ( β)a + β β + θ [ ] = ( β)a + β RβA β + θ = β A + β β RB + ( β) + βθ θ = θ = β β RB + ( β) + βθ β R + β β + ( β) ( β) ( β) 2 V (A) A A A = h(a) h [ V (A) = A h(a) ] + βv (h(a)) R V (A) = c [ ] h (A) R = c + [ cr + βv (A ) + βv (h(a))h (A) ] h (A) k 0

22 U(c 0, c,..., c ) = βt c t y = Ak α A > 0 0 < α < c t + k t+ = Ak α t V (k 0 ) = {c,k } β t c t k = Ak α c V (k 0 ) = c 0,k { c 0 + βv (k )} c 0 = Ak α 0 k k V (k) = k {(Ak α k ) + βv (k )} V (k) = k {(Ak α k ) + βv (k )} V (k) V β < V 0 (k) = 0 V j+ (k) = k {(Ak α k ) + βv j (k )} j

23 V 0 = 0 V (k) = k {(Ak α k ) + β 0} k k 0 k = 0 c = Ak α V (k) = c + β 0 = A + α k V 2 V 2 (k) = k {(Ak α k ) + β[ A + α k ]} Ak α k = αβ k k = αβ + αβ Akα = θ Ak α c = ( θ )Ak α = +αβ Akα V 2 (k) = (c ) + β A + αβ k = ( θ ) + (Ak α ) + β[ A + α θ + α (Ak α )] = ( + αβ) (Ak α ) + β A + [( θ ) + αβ θ ] = ( + αβ) (Ak α ) + ϕ V 3 (k) = k {(Ak α k ) + β[( + αβ) (Ak α ) + ϕ ]} αβ( + αβ) Ak α = k k k = αβ + α2 β 2 + αβ + α 2 β 2 Akα = θ 2 Ak α c = ( θ 2 )Ak α = +αβ+α 2 β 2 Ak α c () Ak α ( + αβ) Ak α ( + αβ + α 2 β 2 ) Ak α j c j = ( + αβ α j β j ) Ak α

24 j 0 < αβ < c = ( αβ)ak α k = αβak α k = (αβa) + α k 0 = ( α)ψ + α k 0 k ψ = α( k 0 ψ) k t ψ = α t ( k 0 ψ) k t = ψ( α t ) + α t k 0 ψ (αβa) α V (k 0 ) β t (c t ) = = = = c t = [A( αβ)] + α k t = [A( αβ)] + αψ( α t ) + α t+ k 0 { β t [A( αβ)] + αψβ t ( α t ) + β t α t+ } k 0 [A( αβ)] β [A( αβ)] β [A( αβ)] β [ + αψ + α (αβa) α + β αβ [ ] + α k 0 αβ β( α) ( β)( αβ) αβ (αβa) ( β)( αβ) + α k 0 αβ ] + α k 0 αβ

25 V (k) = E + F k E F V (k) = k {(Ak α k ) + βe + βf k } Ak α k = βf k k = βf +βf Akα c = +βf Akα V (k) = c + βe + βf k ( ( E + F k = +βf Akα) + βe + βf βf +βf Akα) = A { = +βf + α k + βe + βf AβF A AβF +βf + βe + βf +βf +βf + αβf k } + α( + βf ) k F = α( + βf ) F = α αβ ( β)e = A E = β +βf + βf AβF +βf { [A( αβ)] + } αβ αβ (αβa) + βf = αβ βf +βf = αβ k = βf +βf Akα = αβak α A = A( αβ) +βf c = +βf Akα = ( αβ)ak α V (k) = E + F k = β { [A( αβ)] + } αβ αβ (αβa) + α αβ k

26 x 0 t = 0 E 0 β t r(x t, u t ) x t+ = g(x t, u t, ϵ t+ ) ϵ t P[ϵ t e] = F (e) t ϵ t+ t + u t t t x t x t+j j u t = h(x t ) x = g(x, u, ϵ) E{V (x ) x} = V (x ) df (ϵ) V (x) = {r(x, u) + β E[V u (x ) x]} V (x) r u (x, u) + β E {V (x ) g u (x, u, ϵ) x} = 0 x V (x) = r x [x, h(x)] r u (x, u) + β E {r x (x, u )g u (x, u, ϵ) x} = 0