ECE534, Srig 2018: Fial Exam Problem 1 Let X N (0, 1) ad Y N (0, 1) be ideedet radom variables. variables V = X + Y ad W = X 2Y. Defie the radom (a) Are V, W joitly Gaussia? Justify your aswer. (b) Comute EV W ]. (c) Comute E (V EV W ]) 2]. (d) Defie the radom variable Z = V cw for some costat c R. True or false: V ad Z are deedet radom variables for ay c R. Justify your aswer. Solutio This roblem is similar to HW#3 Problem 1. (a) Sice X, Y are ideedet Gaussia radom variables, they are joitly Gaussia. We further ote that for ay a, b R, we have: av + bw = a(x + Y ) + b(x 2Y ) = (a + b)x + (a 2b)Y, which is Gaussia sice X, Y are joitly Gaussia. Hece, V, W are joitly Gaussia. (b) Usig the corresodig formula for the coditioal mea of joitly Gaussia radom variables, we obtai: EV W ] = EV ] + Cov(V, W ) Cov(V, W ) (W EW ]) = Var(W ) Var(W ) W. We ow comute Cov(V, W ) = E(X + Y )(X 2Y )] = 1 ad Var(W ) = Var(X 2Y ) = Var(X) + 4Var(Y ) = 5. Therefore, EV W ] = 1 5 W. (c) By the class otes, E (V EV W ]) 2] = Var(V ) (Cov(V, W ))2 Var(W ) = 2 ( 1)2 5 = 9 5. (d) First, ote that V, Z are joitly Gaussia radom variables by a similar argumet as i art (a). Also, EZ] = 0. It is easy to comute that for c = 2, Cov(V, Z) = 0, i.e., V, Z are ucorrelated ad therefore ideedet. Hece, the statemet is false. 1
Problem 2 Let X(t), Y (t) be two ideedet rocesses with mea values µ X ad µ Y, resectively. Suose that the corresodig autocorrelatio fuctios are R XX (τ) ad R Y Y (τ), i.e., X(t) ad Y (t) are WSS rocesses. Defie the rocess W (t) = X(t)Y (t). (a) Is W (t) WSS? Justify your aswer. (b) Are W (t) ad X(t) joitly WSS? Justify your aswer. Solutio (a) Sice X, Y are ideedet rocesses we have: Moreover µ W = EW (t)] = EX(t)Y (t)] = EX(t)]EY (t)] = µ X µ Y. R W W (s, t) = EW (t)w (s)] = EX(t)Y (t)x(s)y (s)] = EX(t)X(s)]EY (t)y (s)] = R XX (t s)r Y Y (t s). Sice µ W is time-ideedet ad R W W (t, s) deeds oly o the differece τ = t s, we coclude that W (t) is WSS. (a) W (t) ad X(t) are both WSS rocesses (idividually). To show that they are joitly WSS, we further eed to verify that R W X (s, t) deeds oly o the differece τ = t s. We have: R W X (s, t) = EW (s)x(t)] = EX(s)Y (s)x(t)] = EY (s)]ex(s)x(t)] = µ Y R XX (t s). Therefore, W (t) ad X(t) are joitly WSS rocesses. Problem 3 Let X 1, X 2,... be i.i.d. radom variables with distributio: P (X i = +1) =, P (X i = 1) = q = 1. Defie the radom rocess S with geeric term S = X i ad S 0 = 0. (a) Assume that = q = 1/2. Let Y be a radom rocess with geeric term Y = S, where deotes absolute value. Prove or disrove the followig assertio: Y is a Markov chai. (b) Assume ow that 1/2. Prove or disrove the followig assertio: Y is a Markov chai. 2
Solutio For both arts of this roblem, we use the rovided hits for the fial exam discussed i class. (a) We eed to examie if the rocess Y = S satisfies the Markov roerty, i.e., if P (Y +1 = y +1 Y = y, Y 1 = y 1,..., Y 1 = y 1 ) = P (Y +1 = y +1 Y = y ) or equivaletly if P ( S +1 = s +1 S = s,..., S 1 = s 1 ) = P ( S +1 = s +1 S = s ). We ca see that if S = 0, the ecessarily S +1 = 1 with robability 1. Also, for ay value s > 0 of S, we have that S +1 = s +1 with robability 1/2 ad S +1 = s 1 with robability 1/2. Thus, the kowledge of the sig of S is irrelevat i determiig S +1. Therefore, Y = S is a Markov chai. (b) We eed agai to examie if the rocess Y = S satisfies the Markov roerty whe 1/2. For S = 0, we ecessarily have S +1 = 1 with robability 1. For S = s > 0, we cosider two cases: (i) Suose that S = s > 0. The, S +1 = s +1 with robability ad S +1 = s 1 with robability q. This imlies that S +1 = s +1 with robability ad S +1 = s 1 with robability q. (i) Suose that S = s < 0. The, S +1 = s +1 with robability ad S +1 = s 1 with robability q. This imlies that S +1 = s +1 with robability q ad S +1 = s 1 with robability. The above discussio shows that, whe 1/2, we caot determie the trasitio robabilities of S +1 uambiguously uless we kow the sig of S. I other words, kowledge oly of S is ot eough to determie the trasitio robabilities of S +1. This shows that Y is ot a Markov chai whe 1/2. Problem 4 Cosider the brachig rocess taught i class. here: We briefly summarize the key assumtios X 0 = 1. X t+1 = Z t,1 + Z t,2 + + Z t,xt, where {Z t,i } are i.i.d. radom variables t, i. θ t = P (X t = 0) = P (extictio u to time t). θ = lim t θ t = P (evetual extictio). (a) Suose that {Z t,i } are i.i.d. Ber() radom variables for some 0 < < 1. Show that θ = 1. (a) Assume ow that P (Z t,i = z) = 1/4 for z = 0, 1, 2, 3. Comute θ. 3
Solutio (a) Let Φ Z = E s Z] be the robability geeratig fuctio of each Z t,i, i, t. Sice Z t,i Ber(), we comute: Φ Z (s) = s + (1 ). Let θ t be the robability of extictio by time t. We have show i class that θ t = Φ Z (θ t 1 ), θ 0 = 0. We ow see that θ 1 = Φ Z (θ 0 ) = 1, θ 2 = Φ Z (θ 1 ) = (1 )+(1 ) = 1 2,..., θ t = 1 t. Takig the limit, we have: θ = lim t θ t = lim t ( 1 t ) = 1. Secod Solutio: We ca directly use the mai theorem of the corresodig aalysis from the lecture otes, amely, that θ is the smallest ositive solutio of the fixed oit equatio θ = Φ Z (θ): This equatio has a uique solutio θ = 1. θ = θ + (1 ). Note: θ = 1 is always a solutio of the fixed oit equatio θ = ΦZ (θ) for ay (itegervalued) distributio of Z as we have show i class, sice Φ Z ( θ) = Φ Z (1) = E 1 Z] = 1. The above argumet shows that this solutio is uique whe Z Ber(), 0 < < 1. Third Solutio: The strogest result we ca show is almost sure covergece (imlies also X t 0): t=0 P (X t ɛ) }{{} (i) t=0 EX t ] ɛ = t=0 t ɛ = 1 <, ɛ > 0, ɛ(1 ) where i (i) Markov s iequality is used (X t 0 for all t 0). By the Borel-Catelli Lemma, the evet A t = {X t ɛ} haes oly a fiite umber of times with robability 1. Therefore, P (ω : lim t X t ɛ) = 0 or equivaletly, P (ω : lim t X t = 0) = 1. We coclude that a.s. 0, which i tur imlies that θ = 1. X t 4
(b) For the give robability mass fuctio, Φ Z (s) = 1 4 (1 + s + s2 + s 3 ). As before, θ is the smallest ositive solutio of the fixed oit equatio θ = Φ Z (θ). Settig θ = (1/4)(1 + θ + θ 2 + θ 3 ) results i θ 3 + θ 2 3θ + 1 = 0 or θ 3 θ + θ 2 2θ + 1 = 0 or θ(θ 1)(θ + 1) + (θ 1) 2 = 0 or (θ 1)(θ 2 + 2θ 1) = 0. The roots of this equatio are 1, 2 1, 2 1 ad therefore θ = 2 1. Problem 5 Let X 1, X 2,... be i.i.d. radom variables. Show that where S = X i. EX 1 S, S +1,...] = S, Solutio We first ote that EX 1 S, S +1, S +2,...] }{{} = EX 1 S, X +1, X +2,...] = (i) }{{} (ii) EX 1 S ], where i (i) we use that fact that coditioig with resect to {S, S +1, S +2,...} is equivalet to coditioig with resect to {S, X +1, X +2,...} ad (ii) is due to the ideedece of X 1, X 2,.... Moreover, EX 1 S ] = EX 2 S ] =, EX S ] almost surely by the i.i.d. assumtio o the radom variables X 1, X 2,.... Therefore, usig similar stes as i HW#3 Problem 3 (e): EX 1 S ] = ] EX i S ] = E X i S = ES S ] = S ad the result follows. 5
Secod Solutio: The roblem ca be solved eve without the relacemet of the coditioig i the revious aroach, by oly usig the i.i.d. assumtio. Note that due to the i.i.d. assumtio: EX 1 S, S +1, S +2,...] = EX 2 S, S +1, S +2,...] = = EX S, S +1, S +2,...] almost surely. We ow have: EX 1 S, S +1, S +2,...] = from which the result follows. ] EX i S, S +1, S +2,...] = E X i S, S +1, S +2,... = ES S, S +1, S +2,...] = S Third Solutio: I a similar sirit, we ca eve use Orthogoality Pricile to get the result. Assume that EX1 2 ] <. Based o ay of the revious aroaches, we ca either show that ( X 1 S ) f(s ) for ay Borel-measurable fuctio f such that Ef(S ) 2 ] < or ( X 1 S ) f(s, S +1,...) for ay Borel-measurable fuctio f such that Ef(S, S +1,...) 2 ] <. Focusig for simlicity o the first case, we have: ( E X 1 S ) ] ( f(s ) = E X 1 X ) ] f(s ) = 1 EX 1f(S )] 1 EX i f(s )] = 1 EX 1f(S )] 1 EX 1f(S )] = 0, where due to the i.i.d. assumtio EX 1 f(s )] = EX i f(s )], 2 i. i=2 Problem 6 Cosider two radom sequeces {X } 0 ad {Y } 0 covergig i robability to the limits X ad Y, resectively: X X ad Y Y. (a) Show that for ay costats a, b R. ax + by (b) Let g(x) be a cotiuous fuctio. Show that ax + by g(x ) g(x). Note: Recall that g : D R is cotiuous at x D if ɛ > 0, δ > 0 such that y D : x y < δ g(x) g(y) < ɛ. 6
Solutio We first write exlicitly the defiitio of covergece i robability for both sequeces: lim P ( X X > ɛ) = 0 ad lim P ( Y Y > ɛ) = 0, ɛ > 0. (a) If a = 0 or b = 0 or a = b = 0 the the result is immediate. Suose that a 0 ad b 0. Usig similar stes as i HW#2 Problem 3 (a), we have: P ( ax + by ax by > ɛ) P ( a X X + b Y Y > ɛ) for ay ɛ > 0. Therefore, ax + by ax + by. P ( a X X > ɛ/2) + P ( b Y Y > ɛ/2) (b) Sice g(x) is cotiuous, this imlies that ɛ > 0, δ > 0 such that x y < δ g(x) g(y) < ɛ. By the rovided hits for the exam, the imlicatio results i P ( X X < δ) P ( g(x ) g(x) < ɛ). 0 Therefore, P ( g(x ) g(x) > ɛ) P ( X X > δ) 0. Sice the result holds for ay ɛ > 0, we coclude that g(x ) g(x). Problem 7 Let X be a radom variable such that EX 2 ] <. Suose that we wat to estimate X by observig a sequece of radom variables Y 1, Y 2,.... Let ˆX = EX Y 1, Y 2,..., Y ] = EX Y ] be the MMSE estimator of X give the observatios Y 1, Y 2,..., Y. Defie MMSE = E (X ˆX ) 2]. Show the followig: ] ] (a) E ˆX2 E ˆX2 +1 ] ad E ˆX2 E X 2],. (b) MMSE MMSE +1 for ay 1. 7
Solutio (a) For the first art, we have: ] E ˆX2 = E E X Y ] 2] = }{{} (i) = E E X Y +1] ] 2 = E ˆX +1]. 2 E E E X Y +1] Y ] ] 2 }{{} (ii) I (i), the smoothig roerty of coditioal exectatio is used. I (ii), Jese s iequality is alied. For the secod art, ] E ˆX2 = E E X Y ] 2] I (iii), Jese s iequality is emloyed. }{{} (iii) E E X 2 Y ]] = EX 2 ]. E E E X Y +1] ]] 2 Y Secod Solutio for the secod art: From the course otes we kow that the MMSE error is E (X ˆX) ] 2 = E X 2] ] E ˆX2. This formula was derived i class based o the Orthogoality ] Pricile. Usig this formula ad the fact that MMSE 0, we obtai E ˆX2 E X 2], 1. ] (b) Alyig agai the formula for MMSE, the result is a immediate cosequece of E ˆX2 ] E ˆX2 +1 from art (a): MMSE = E X 2] E (EX Y ]) 2] E X 2] E (EX Y +1 ]) 2] = MMSE +1. 8