Final Examination Solutions (Total: 100 points)

Transcription

1 Final Examination Solutions (Total: points) There are 4 problems, each problem with multiple parts, each worth 5 points. Make sure you answer all questions. Your answer should be as clear and readable as possible.. Consider the random variables X and Y with the joint pdf { c if x y, y f X,Y (x, y) otherwise, where c is a constant. (a) Find c (b) Find E(X), E(Y ) and E(XY ). (c) Given Y, find the MMSE estimate of X, ˆX g(y ). (d) Find the MMSE linear estimate of X given Y, X ay + b. (e) Find the MSE of ˆX and compare to the MSE of the best linear estimator X. Solution: This problem is similar to problem in HW3 where we consider the right half of the diamond distribution. (a) This is the right half of the diamond distribution in HW3 problem. Hence, the area is. This implies c.

2 (b) For x, f X (x) x +x ( ) c dy 4 x. Similarly for y, Now, f Y (y) y ( ) c dx y. and E[Y ]. E[X] f X (x) dx ( ) x 4 x dx [ ] x 4 x3 3 f Y (y) dy 3, ( ) y ( ) y y dy + + y dy Also, E[XY ]. ( x dx +x xy dy ) dx (c) For y and x, f X Y (x) f X,Y (x, y) f Y (y) ( ) y

3 Hence, E[X Y y] y xf X Y (x) dx y. The MMSE estimator ˆX is given by Y. (d) The linear MMSE is given by ˆX L Cov(X,Y ) (Y E(Y ))+E(X). Now, Cov(X, Y ) σy E(XY ) E(X)E(Y ). This implies ˆX L E(X) 3. (e) MMSE is given by E Y (Var(X Y )) E(X ) E(E(X Y ) ) and E(E(X Y ) ) + E(Y ) 8 4 E( Y ). Now, E(X ) [ x 3 4. ( ) x 4 x 3 dx 3 x4 4 Similarly, we find E(Y ) and E( Y ) 3. Hence, E Y (Var(X Y )). 48 The linear MSE is given by σx E(X ) (E(X)). Consider a communication channel best modeled by a binary symmetric channel (see Figure ) with cross over probability ɛ < ɛ where the value of ɛ is not known. Let us assume ɛ is a Uniform random variable between and. To learn the value of ɛ, we transmit a bunch of bits (this in communication is called training and is done to learn a channel) and received the received bits. (a) What is the probability that the first three bits are received as (flipped by the channel)? (b) What is the probability that the second bit is received as, given that the first bit was received as? (c) If you transmit bits of s and receive bits in error (receive bits), what can you say about ɛ? Can you characterize a bound on how well your estimate works? Solution: This problem is similar to problem 4 of HW3. Let Y i denote the i-th bit received. ] 36. 3

4 (a) P(Y Y Y 3 ) 4. P(Y Y Y 3 ɛ)f(ɛ) dɛ ɛ 3 dɛ (b) P(Y Y ) P(Y Y ) P(Y ) ɛ dɛ ɛ dɛ 3. (c) Let X denote the number of s received, i.e. X n i Y i. There are two ways to solve this problem. Using Chebychev s inequality: Let ˆɛ n n i Y i n denote the empirical mean of number of s received and we have E(ˆɛ n ɛ) ɛ and Var(ˆɛ n ɛ) σ Y n (ɛ) ɛ( ɛ). In this case, we use the n empirical mean for bits i.e. ˆɛ. as an estimate of ɛ and calculate the probability of ˆɛ ɛ being bounded some δ >. For this we use the Chebyshev inequality as follows: P{ ˆɛ E(ˆɛ ɛ) δ ɛ} P{ ˆɛ ɛ δ ɛ} Var(ˆɛ ɛ) δ σ Y (ɛ) δ. Now choose δ kσ Y (ɛ). Therefore, ˆɛ ɛ kσ Y (ɛ) with probability at least. For instance, if k then ˆɛ k ɛ σ Y (ɛ) with probability at least.99, meaning for a fixed ɛ, the estimate ˆɛ is within one standard deviation of ɛ with probability.99. Using MMSE: In this case, we use MMSE estimate of ɛ given observation X, i.e. ˆɛ E(ɛ X) as an estimate of ɛ and calculate MMSE error. For X x, we have ˆɛ(x) E(ɛ X x). To find the MMSE estimate we find the posterior distribution of 4

5 ɛ given X, Hence, f ɛ X (e x) P X ɛ(x e)f(e) P X (x) P X ɛ (x e)f(e) P X ɛ(x e )f(e ) de e x ( e) ( x) e x ( e ) ( x) de e ( e) 9 f ɛ (e ) e ( e ) 9 de. This is Beta distribution with parameters α and β 9, where pdf of Beta distribution is given by B(α,β) xα ( x) β and B(α, β) is the Beta function which forms the normalizing constant. Hence, for X x, ˆɛ(x) E(ɛ X x) the MMSE estimate of E given X is The MSE for the estimator is ˆɛ E(ɛ X) X + ɛf E x (ɛ x) dɛ x +, n i Y i +. E X (Var(ɛ X)) E(ɛ ) E(E(ɛ X) ) ( (X ) ) + 3 E 3 E(X ) + E(X) + 3 Var(X) + E(X) + E(X) + 3 Var(X) + (E(X) + ). Now, X ɛ has a binomial distribution with parameters (, ɛ). Hence, E(X) E(E(X ɛ)) ɛdɛ 5 and by law of conditional variances Var(X) E(Var(X ɛ)) + Var(E(X ɛ)) E(ɛ( ɛ)) + Var(ɛ) 85. 5

6 3. Let Therefore, the MMSE is E X (Var(ɛ X)).6. Note: Using the expression for ˆɛ() we have and ˆɛ().784, Var(ɛ X ).96. Note that empirical estimate ˆɛ is very close to the MMSE estimate ˆɛ(). X n Z n + Z n, n,,..., where Z, Z, Z,... are i.i.d. N(, ). Let Y n X n + X n, n, 3,.... (a) Find the mean and autocorrelation functions of {Y n }. (b) Is {Y n } wide-sense stationary? Justify your answer. (c) Is {Y n } strict-sense stationary? Justify your answer. (d) Is {Y n } Markov? (e) Find E(Y Y 3 ) and E(Y Y 3, Y 4 ). (f) Find E(Z Y ). Solution: This is similar to additional problem of HW7. (a) Note that Y n Z n + Z n + Z n. The mean function µ n E(Y n ). The autocorrelation function 6 n m, 4 n m or n m +, R Y (n, m) n m or n m +, otherwise. (b) Since the mean and autocorrelation functions are time-invariant, the process is WSS. (c) Since (Y,..., Y n ) is a linear transform of a GRV (Z, Z,..., Z n ), the process is Gaussian. Since the process is WSS and Gaussian, it is SSS. (d) No, it is not Markov. See part (e) for justification. 6

7 (e) Since the process is Gaussian, the conditional expectation (MMSE estimate) is linear. Hence, E(Y Y 3, Y 4 ) [ 4 ] [ ] [ ] 6 4 Y3 4 6 (Y 3 Y 4 ). Similarly, E(Y Y 3 ) (/3)Y 3. Since E(Y Y 3, Y 4 ) E(Y Y 3 ), the process is not Markov. (f) E(Z Y ) σ Z Y σ Y Y and σ Z Y. Therefore, E(Z Y ) 6 Y. Y 4 7

8 Useful Facts: Let X be a uniform random variable and Y is a random variable that has Beta distribution with parameters α > and β >, then f X (x) { b a if a x b otherwise and f Y (y) { B(α,β) yα ( y) β if < y < otherwise where B(α, β) denotes the Beta function which normalizes the distribution. MMSE estimate of X given Y is given by ˆX E(X Y ); its MSE is given by E(Var(X Y )). The MMSE linear estimate of X given Y is given by X Cov(X,Y ) (Y E(Y )) + E(X). σy Let Z max{x, Y }, X and Y independent. Then, F Z (z) F X (z)f Y (z). Let Z AX and X is a Gaussian random vector with mean µ and covariance matrix Σ. Then Z is a Gaussian random vector (jointly Gaussian with X) with mean Aµ and covariance matrix AΣA T. Recall the following table Random Variable Mean Variance Bern(p) p p( p) Geom(p) p p p Binom(n, p) np np( p) U[ a, b ] a+b (b a) Beta(α, β) α α+β αβ (α+β) (α+β+) N ( µ, σ ) µ σ 8