PoissonprocessandderivationofBellmanequations

Transcription

1 APPENDIX B PoissonprocessandderivationofBellmanequations 1 Poisson process Let us first define the exponential distribution Definition B1 A continuous random variable X is said to have an exponential distribution with parameter λ> if its cumulative distribution function (cdf) is given by: F (t) =P {X <t} =1 e λt for all t This implies that the probability density function (pdf) is equal to: f(t) =λe λt for all t The exponential distribution has mean Z+ E [X] = tf(t)dt = 1 λ and variance V [X] =E X 2 (E [X]) 2 = 1 λ 2 It is straightforward to show the following proposition: Proposition B1 The exponential distribution with parameter (or rate) λ> has the following properties: (i) Independence: P {X >t+ t } = P {X >t} P {X >t } (ii) Memoryless P {X >t+ t X>t} = P {X >t } t, t Observe that the exponential distribution is the unique distribution possessing the memoryless property Following Ross (1996), let us now define a counting process (and its properties) and then a Poisson process 395

2 396 B POISSON PROCESS AND DERIVATION OF BELLMAN EQUATIONS Definition B2 Let {X n, n 1} be a sequence of random variables representing the inter-event times Define S =, S n = X X n ThenS n is the time of occurrence of the nth event Define S(t) =max{n S n t}, t Thus S(t) represents the total number of events that have occurred up to time t A stochastic process {S(t), t } is called a counting process Definition B3 A counting process {S(t), t } is said to possess: (a) independent increments if the number of events which occur in disjoint intervals are independent, ie for t 1 t n, the increments S(t 1 ), S(t 2 ) S(t 1 ),, S(t n ) S(t n 1 ) are independent random variables (b) stationary increments if the distribution of the number of events which occur in any interval of time only depends the length of the time interval, ie the distribution of X(t + t ) X(t ) is independent of t A Poisson process is frequently used as a model for counting events occurring one at a time Let us now define formally a Poisson process Definition B4 The counting process {S(t), t } is said to be a Poisson process having rate λ>, if: (i) The process has stationary and independent increments; (ii) The number of events in any interval of length t is Poisson distributed with mean λt Thatis,forallt, λt (λt)n P {S(t) =n} = e, n! n =, 1, (B1) We have the following alternative definition: 1 Definition B5 Thecounting process{s(t), t } with S() = is said to be a Poisson process having rate λ>, if: (i) The process has stationary and independent increments; 1 A function f : R R is said to be an o(h) function if f(h) lim h h =

3 (ii) P {S(h) =1} P {S(t + h) S(t) =1} = λh + o(h); (iii) P {S(h) 2} P {S(t + h) S(t) 2} = o(h) 1 POISSON PROCESS 397 This definition helps us understand how randomness in time can be interpreted Condition (i) implies that what happens under non-overlapping time intervals is independent Condition (ii) states that rate λ is constant over time while condition (iii) says that two (or more) eventscannot occuratthesametime Ross (1996) shows that these two definitions (ie definitions B4 and B5) are equivalent Example B1 Consider a person who receives a lot of junk mails in his/her mailbox Assume that the number of junk mails follows a Poisson process having rate 2 per hour B11 What is the probability that no junk mail arrives between 9 am and 11 am? This probability is defined by P {S(11) S(9) = } UsingDefinition B4 and in particular equation (B1), we obtain (for λ =2): 2t (2t) P {S(11) S(9) = } = e! = e 2 2 = 183 Observe that, because of time homogeneity, this probability is the same as the probability that no junk mail arrives between 2: pm and 4: pm or any other times as long as there aretwohoursdifference B12 What is the expected number of junk mails that arrive during 8 hours? The expected number of junk mails that arrive during 8 hours is: E [S(8)] = λ 8=16 B13 What is the probability that one junk mail arrives between 1: pm and 2: pm and two junk mailsarrivebetween1:3pmand2:3pm? Using time homogeneity and using hours as units of time, let us start at t =(which corresponds to 1: pm here) so that 2: pm corresponds to t =1(here, 1 hour corresponds to 1 unit of time) Then, 1:3 pm corresponds to t =5 and 2:3 pm to t =15 Thus, the probability that one junk mail arrives between 1: pm and 2: pm and two junk mails arrive between 1:3 pm and 2:3 pm can be written as: P {S(1) = 1, S(15) S(5) = 2}

4 398 B POISSON PROCESS AND DERIVATION OF BELLMAN EQUATIONS Using the fact that increments over (, 5], (5, 1], and(1, 15] are independent, we can write the following: P {S(1) = 1, S(15) S(5) = 2} P = 1 P {S(5) = n, S(1) S(5) = 1 n, S(15) S(1) = 1 + n} n= Let us explain this last equation We divide time in three intervals: (, 5], (5, 1], and (1, 15], that is between 1: pm and 1:3 pm, 1:3 pm and 2: pm, 2: pm and 2:3 pm Then, in order to have 1 junk mail between t =(ie 1: pm) and t =1(ie 2: pm), it has to be that either no junk mail arrives between t =and t =5 (ie 1:3 pm) and 1 arrives between t =5 and t =1or 1 junk mailarrivesbetweent =and t =5 and arrives between t = 5 and t = 1 Similarly, in order to have 2 junk mails between t = 5 and t =15 (ie 2:3 pm), it has to be that either no junk mail arrives between t =5 and t =1and 2 arrive between t =1and t =15 or 2 junk mails arrive between t =5 and t =1and arrives between t =1and t =15 or 1 junk mail arrives between t =5 and t =1and 1 arrives between t =1and t =15 By combining all these possibilities, it hastobethat: P {S(1) = 1, S(15) S(5) = 2} = P {S(5) =, S(1) S(5) = 1, S(15) S(1) = 1} +P {S(5) = 1, S(1) S(5) =, S(15) S(1) = 2} In other words, either no junk mail arrives between t =and t =5 and 1 junk mail arrives both between t =5 and t =1and between t =1and t =15, or 1 junk mail arrives between t =and t =5, junk mailbetweent =5 and t =1and 2 between t =1and t =15

5 1 POISSON PROCESS 399 We have: 1P P {S(5) = n, S(1) S(5) = 1 n, S(15) S(1) = 1 + n} n= P = 1 P {S(5) = n} P {S(1) S(5) = 1 n} P {S(15) S(1) = 1 + n} n= P = 1 1 (1)n (1)1 n (1)1+n e e 1 e 1 n= n! (1 n)! (1 + n)! = e 3 P 1 1 n= n!(1 n)! (1 + n)! µ = e =15e 3 ' Consider a Poisson process {S(t), t } and denote the time of the first event by T 1 Moreover, for n>1, lett n denote the elapsed time between the (n 1)th and the nth event The sequence {T n,n=1, 2, } is called the sequence of interarrival times We can now determine the distribution of the T n We have the following result: Proposition B2 T n, n =1, 2, are independent identically distributed (iid) exponential random variables having mean 1/λ This means, in particular, that and P {T 1 >t} = P {S(t) =} = e λt P {T 2 >t T 1 = t } = e λt etc Furthermore, the arrival time of the nth event, also called the waiting time until the nth event and denoted by Σ n, is given by P Σ n = n T i, n 1 i=1 Σ n has a gamma distribution with parameters n and λ That is, the probability density of Σ n is: f Σn (t) =λe λt (λt)n 1 (n 1)!

6 4 B POISSON PROCESS AND DERIVATION OF BELLMAN EQUATIONS 2 An intuitive way of deriving the Bellman equations Let us first derive the expected lifetime-utility of an unemployed worker I U This analysis is valid for both search-matching (Part 1) and efficiency wage (Part 2) models During a small interval of time dt, the unemployed worker obtains W U and during this time, he/she may find a job and enjoys an expected lifetime-utility level of I E at time t + dt The probability that he/she finds a job is: adt + o(dt) with lim o(dt)/dt =Ifhe/shedoesnotfind a job, dt then he/she enjoys utility I U at time t + dt Observe that, during this small interval of time dt, the probability that the unemployed worker leaves unemployment and then straightaway loses his/her new job is negligible with respect to dt (sinceitisatermin(dt) 2 ) We have: 2 I U (t) =W U (t)dt + This is equivalent to 1 1+rdt [adt I L(t + dt)+(1 adt)i U (t + dt)] (1 + rdt) I U (t) =(1+rdt) W U (t)dt + adt I L (t + dt)+(1 adt)i U (t + dt) ri U (t)dt = W U (t)dt + rw U (t)(dt) 2 + adt [I L (t + dt) I U (t + dt)] + I U (t + dt) I U (t) Now, by dividing everything by dt, we obtain (observe that (dt) 2 is negligible compared to dt) ri U (t) =W U (t)+a[i L (t + dt) I U (t + dt)] + I U(t + dt) I U (t) dt By taking the limit when dt, weget ri U (t) =W U (t)+a [I L (t) I U (t)] + I U (t) since I U (t) di U(t) I U (t + dt) I U (t) = lim dt dt dt In steady-state, I U (t) =and I U (t) =I U and I L (t) =I L The steady-state lifetime expected utility of an unemployed worker is given by: 2 Wecouldhavediscountedinadifferent way, ri U = W U + a (I L I U ) 1 I U (t) = 1+rdt [W U (t)dt + adti E (t + dt)+(1 adt)i U (t + dt)] ie before starting the period In this case, it is easily verified that the Bellman equation would have been exactly the same

7 3 A FORMAL WAY OF DERIVING THE BELLMAN EQUATIONS 41 Let us now determine the expected lifetime-utility of a non-shirking employed worker I L We have: I L (t) =WL NS 1 (t)dt + δdt IU (t + dt)+(1 δdt)il NS (t + dt) 1+rdt By replicating the same analysis as for the unemployed, we easily obtain for employed workers: ri L = W NS L δ (I E I U ) Letusnowfocusmorespecifically on efficiency wage models (Part 2) Determining the case of shirking is more complicated since shirking workers can lose their jobs either by an exogenous shock δ or by being caught shirking at rate m Wehave IL(t) S =WL S 1 (t)dt + δdtiu (t + dt)+mdt I U (t + dt)+(1 (δ + m) dt) I S 1+rdt L(t + dt) By replicating the same analysis, we obtain: ri S L = W S L (δ + m) I S L I U (B2) 3 A formal way of deriving the Bellman equations As stated above, changes in employment status are assumed to be governed by a Poisson process with two states: employed or unemployed Consider a Poisson process (as defined by Definition B4 or Definition B5) As shown in Proposition B2, the key feature of these stochastic processes is that the duration time spent in each state is a random variable with exponential distribution More precisely, if we denote the (random) unemployment and employment duration times by T a and T δ,then F (T a )=P {T a <t} =1 e at a F (T δ )=P {T δ <t} =1 e δt δ This implies that the probability densities are given by: f(t a )=ae ata (B3) f(t δ )=δe δt δ (B4)

8 42 B POISSON PROCESS AND DERIVATION OF BELLMAN EQUATIONS As a result, the average time spent in each state is equal to (see Definition B1): E [T a ] = = = Z + Z + Z + = 1 a = 1 a T a f(t a )dt a at a e ata dt a e ata dt a T a e ata + e at a + T a e ata + since Similarly, lim T a + T a e at a = lim Ta T a e at a = E [T δ ]= Z + T δ f(t δ )dt δ = 1 δ As above, let us first determine the expected lifetime-utility of an unemployed worker, I U It is given by: Z Ta I U = E Ta W U e rt dt + e rta I L (B5) I U is thus the discounted value at time t = The unemployed worker remains unemployed during a random period of time T a During this period, he/she earns w U discounted at rate r Then, after this period T a, he/she becomes employed and obtains an expected utility of I L discounted at rate r starting at time t = T a By developing (B5), we obtain: I U = + Z Z Ta + Z W U e rt dt f(t a )dt a + e rt a I E f(t a )dt a Since Z Ta W U e rt dt = W U 1 e rt a r,

9 3 A FORMAL WAY OF DERIVING THE BELLMAN EQUATIONS 43 and using (B3), it can be rewritten as: I U = a W U r = a W U r + Z + Z 1 e rt a e at a dt a + ai L e rt a e at a dt a + Z e at a e Z (r+a)t a dt a + ai L = W U r + a + a r + a I L We finally obtain: ri U = W U + a (I L I U ) + e (r+a)t a dt a (B6) Similarly, we can calculate the expected lifetime-utility of an employed worker I L (non shirking in the efficiency wage model) It is: Z Tδ IL NS = E Tδ WL NS e rt dt + e rt δ I U (B7) By making exactly the same analysis, one easily obtains: ri NS L = W NS L δ (I L I U ) (B8) For the efficiency wage model, we need to consider the case of a shirker It is more complicated since, when employed, a shirker can lose his/her job because either he/has been caught shirking or the job has been destroyed Denote by T m the(random)lengthoftimeuntilthe next control of shirking occurs This implies that T a is still the (random) unemployment duration time whereas min(t δ,t m ) is now the employment duration time for a shirker Since we know (see, for example, Kulkarni, 1995, ch 5) that min(t δ,t m ) is a random variable characterized by an exponential distribution of parameter δ + m, ie F (min(t δ,t m )) = P [min(t δ,t m ) <t]=1 e (δ+m) min(t δ,t m) then the expected lifetime-utility of a shirker IL S is equal to: " Z # min(tδ,t m) IL S = E WL S e rt dt + e r min(t δ,t m ) I U By doing exactly the same kind of manipulations as above, we obtain (B2)