6.231 Dynamic Programming and Optimal Control Midterm Exam, Fall 2011 Prof. Dimitri Bertsekas

Transcription

1 6.231 Dyamic Programmig ad Optimal Cotrol Midterm Exam, Fall 2011 Prof. Dimitri Bertsekas Prolem 1: (50 poits) Alexei plays a game that starts with a deck with lack cards ad r red cards. Alexie kows ad r. At each time period he draws a radom card ad decides etwee the followig two optios: (1) Without lookig at the card, predict that it is lack, i which case he wis the game if the predictio is correct ad loses if the predictio is icorrect. (2) Discard the card, after lookig at its color, ad cotiue the game with oe card less. If the deck has oly lack cards he wis the game, while if the deck has oly red cards he loses the game. Alexei wats to fid a policy that maximizes his proaility of a wi. (a) Formulate Alexei s prolem ito the format of the fiite-horizo asic prolem with perfect state iformatio. Idetify states, cotrols, disturaces, etc. () Write the DP algorithm. Use iductio to show that the optimal proaility of a wi startig with lack cards ad r red cards is. + r (c) Characterize the optimal policies. (d) Suppose that Alexei is give the additioal optio to radomize his decisio at each time period. I particular, he may choose a proaility p [0, 1], flip a coi that has proaility of head equal to p, ad decide upo optio 1 or 2 aove depedig o the outcome of the flip. What would the e the optimal policies? Solutio: (a) The state is the curret pair (, r) plus a termiatio state, to which we move upo selectig optio 1. If optio 2 is selected, the we move to state ( 1, r) with proaility /(+r) ad to state (, r 1) with proaility r/( + r). At states with either = 0 or r = 0, oly optio 1 is availale. The reward per stage is 1 if Alexei chooses optio 1 ad wis the game, ad 0 otherwise. (,c,d) The DP algorithm is J (, r) = [ mi p { + r p 0,1} ( + (1 p) + r J ( 1, r) + r J (, r 1) + r 1 )],

2 where p = 0 ad p = 1 correspod to optios 1 ad 2, respectively. We show y iductio that the optimal reward-to-go startig with lack ad r red cards is J (, r) =, + r ad that all strategies give the same proaility of a wi. This formula is correct for all (, r) with + r = 1, ased o the assumptio that Alexei wis whe the deck has oly lack cards ad loses if the deck has oly red cards. Assumig that the formula is correct for all (, r) with + r = k 1, we will show that it is true for all (, r) with + r = k. For ay p [0, 1], the right-had side of the DP algorithm is ( ) r p + (1 p) J ( 1, r) + J (, r 1) + r + r + r ( ) 1 r = p + (1 p) + + r + r + r 1 + r + r 1 = p + (1 p) + r + r =, + r where the secod equality makes use of the iductio hypothesis. Thus J (, r) =, + r ad the iductio is complete. For the case p [0, 1], this calculatio also yields the same formula for J, regardless of the value of p chose. Prolem 2: (50 poits) A amitious egieer, curretly employed i a recetly impoverished coutry at a salary c, seeks employmet at aother coutry. He receives a jo offer at each time period, which she may accept or reject. The offered salary takes oe of possile values w 1,..., w with give proailities ξ 1,..., ξ, idepedet of precedig offers. If she accepts the offer, she keeps the jo for the rest of her life. If she rejects, she cotiues at her curret salary c for oe more period ad is eligile to accept offers i future periods. Assume that icome is discouted y a factor α < 1. (a) Formulate the egieer s prolem as a discouted ifiite horizo prolem. Idetify states, cotrols, disturaces, etc. () Formulate the Bellma equatio ad characterize the optimal policy. 2

3 (c) Cosider the variat of the prolem where there is a give proaility p i that the worker will e fired from her jo at ay oe period if the salary is w i. Oce fired, she returs to her ow coutry ad resumes her former employmet ad is eligile to accept offers i future periods. Formulate the Bellma equatio ad characterize the optimal policy. (d) Do part (c) for the case whe icome is ot discouted ad the worker maximizes the average icome per period. (e) Assume that for the jo with salary w i, the expected umer of periods for keepig that jo is t i, ad afterwards she returs to her former employmet as i part (c). Formulate the Bellma equatio for the prolem of maximizig her average retur usig t i istead of p i. Characterize the optimal policy. Solutio: (a) Let the states e s i, i = 1,...,, correspodig to the worker eig uemployed ad eig offered a salary w i, ad s i, i = 1,...,, correspodig to the worker eig employed at a salary level w i. Suppose the curret state is s i, the cotrol is to either accept ad go to state s i, or to reject ad wait for the ext offer. The reward is w i if she accepts, or c if she rejects. Suppose the state is s i, there is o more decisio to make ad the reward is always w i. () The Bellma equatio is J(s i ) = max c + α ξ j J(s j ), w i + αj(s i), i = 1,...,, (1) From Eq. (2), we have J(s i) = w i + αj(s i), i = 1,...,. (2) w i J(s i) = i = 1,...,, 1 α so that from Eq. (1) we otai J(s i ) = max c + α ξ j J(s j ), w i 1 α, Thus it is optimal to accept salary w i if w i (1 α) c + α ξ j J(s j ). 3

4 The right-had side of the aove relatio gives the threshold for acceptace of a offer. (c) I this case the Bellma equatio ecomes J(s i ) = max c + α ξ j J(s j ), w i + α (1 p i )J(s i) + p i ξ j J(s j ) J(s i) = w i + α (1 p i )J(s i) + p i ξjj( s j ). (4) Let us assume without loss of geerality that w 1 < w 2 < < w. Let us assume further that p i = p for all i. From Eq. (4), we have so it follows that w i + p J(s i) = ξj J(s j ), 1 α(1 p) (3) J(s 1) < J(s 2) < < J(s ). (5) We thus otai that the secod term i the maximizatio of Eq. (3) is mootoically icreasig i i, implyig that there is a salary threshold aove which the offer is accepted. I the case where p i is ot idepedet of i, salary level is ot the oly criterio of choice. There must e cosideratio for jo security (the value of p i ). However, if p i ad w i are such that Eq. (5) holds, the there still is a salary threshold aove which the offer is accepted. (d) The Bellma equatio has the form λ+h(s i ) = max c + ξ j h(s i ), w i + (1 p i )h(s i) + p i ξ j h(s j ), i = 1,...,, (3) λ + h(s i) = w i + (1 p i )h(s i) + p i ξ j h(s j ), i = 1,...,. (4) From these equatios, we have λ + h(s i ) = max c + ξ j h(s j ), λ + h(s i), i = 1,...,, 4

5 so it is optimal to accept a salary offer w i if h(s i) is o less that the threshold c λ + ξ j h(s j ). Here λ is the optimal average salary per period (over a ifiite horizo). If p i = p for all i ad w 1 < w 2 < < w, the from Eq. (4) it follows that h(s i) is mootoically icreasig i i, ad the optimal policy is to accept a salary offer if it exceeds a certai threshold. (e) Let us deote y i the state correspodig to the offer (w i, t i ). We have the followig Bellma s equatio: h(i) = max wi t i λt i + p j h(j), c λ + p j h(j) The optimal policy is to accept offer (w i, t i ) if c w i λ, t i where λ is the optimal average icome per uit time. 5

6 MIT OpeCourseWare Dyamic Programmig ad Stochastic Cotrol Fall 2015 For iformatio aout citig these materials or our Terms of Use, visit: