UCSD ECE153 Handout #34 Prof. Young-Han Kim Tuesday, May 27, Solutions to Homework Set #6 (Prepared by TA Fatemeh Arbabjolfaei)

Transcription

1 UCSD ECE53 Handout #34 Prof Young-Han Kim Tuesday, May 7, 04 Solutions to Homework Set #6 (Prepared by TA Fatemeh Arbabjolfaei) Linear estimator Consider a channel with the observation Y XZ, where the signal X and the noise Z are uncorrelated Gaussian random variables Let E[X], E[Z], σ X 5, and σ Z 8 (a) Find the best MSE linear estimate of X given Y (b) Suppose your friend from Caltech tells you that he was able to derive an estimator with a lower MSE Your friend from UCLA disagrees, saying that this is not possible because the signal and the noise are Gaussian, and hence the best linear MSE estimator will also be the best MSE estimator Could your UCLA friend be wrong? (a) We know that the best linear estimate is given by the formula ˆX Cov(X,Y) σy (Y E(Y))+E(X) Note that X and Z Gaussian and uncorrelated implies they are independent Therefore, E(Y) E(XZ) E(X)E(Z), E(XY) E(X Z) E(X )E(Z) (σ X +E (X))E(Z), E(Y ) E(X Z ) E(X )E(Z ) (σ X +E (X)),(σ Z +E (Z)) 7, Cov(X,Y) σ Y σ Y E(Y ) E (Y) 68, E(XY) E(X)E(Y) σ Y Using all of the above, we get 5 34 ˆX 5 34 Y + 7 (b) The fact that the best linear estimate equals the best MMSE estimate when input and noise are independent Gaussians is only known to be true for additive channels For multiplicative channels this need not be the case in general In the following, we prove Y is not Gaussian by contradiction Suppose Y is Gaussian, then Y N(,68) We have f Y (y) π 68 e (y ) 68

2 On the other hand, as a function of two random variables, Y has pdf f Y (y) ( y ) X (x)f Z dx f x But these two expressions are not consistent, because ( ) 0 f Y (0) X (x)f Z dx f Z (0) f x e (0 ) 8 π 8 π 68 e (0 ) 68 f Y (0), f X (x)dx f Z (0) which is a contradiction Hence, X and Y are not joint Gaussian, and we might be able to derive an estimator with a lower MSE Additive-noise channel with path gain Consider the additive noise channel shown in the figure below, where X and Z are zero mean and uncorrelated, and a and b are constants Z X a b Y b(ax +Z) Find the MMSE linear estimate of X given Y and its MSE in terms only of σ X, σ Z, a, and b By the theorem of MMSE linear estimate, we have ˆX Cov(X,Y) σy (Y E(Y))+E(X) Since X and Z are zero mean and uncorrelated, we have E(X) 0, E(Y) b(ae(x)+e(z)) 0, Cov(X,Y) E(XY) E(X)E(Y) E(Xb(aX +Z)) abσx, σy E(Y ) (E(Y)) E(b (ax +Z) ) b a σx +b σz Hence, the best linear MSE estimate of X given Y is given by ˆX aσ X ba σx Y +bσ Z

3 3 Image processing A pixel signal X U[ k,k] is digitized to obtain X i+, if i < X i+, i k, k +,, k, k To improve the the visual appearance, the digitized value X is dithered by adding an independent noise Z with mean E(Z) 0 and variance Var(Z) N to obtain Y X +Z (a) Find the correlation of X and Y (b) Find the best linear MSE estimate of X given Y Your answer should be in terms only of k, N, and Y (a) From the definition of X, we know P{ X i+ } P{i < X i+} k By the law of total expectation, we have Cov(X,Y) E(XY) E(X)E(Y) E(X( X +Z)) E(X X) k E[X X i < X i+]p(i < X i+) i k k i+ i k i 4k Since, k i i k(k +)(k +)/6 (b) We have x(i+ ) k dx 8k k (i+) 4k i k k (i ) i E(X) 0, E(Y) E( X)+E(Z) 0, σ Y Var X +VarZ k (i+ ) k +N k (i+) +N 4k 4k i k Then, the best linear MMSE estimate of X given Y is given by i0 ˆX Cov(X,Y) σy (Y E(Y))+E(X) 4k 4k +N Y 4k 4k +N Y +N 4 Covariance matrices Which of the following matrices can be a covariance matrix? Justify youranswereitherbyconstructingarandomvectorx, asafunctionoftheiidzeromeanunit variance random variables Z,Z, and Z 3, with the given covariance matrix, or by establishing a contradiction 3

4 [ (a) 0 ] (b) [ ] (c) 3 (d) (a) This cannot be a covariance matrix because it is not symmetric (b) This is a covariance matrix for X Z +Z and X Z +Z 3 (c) This is a covariance matrix for X Z, X Z +Z, and X 3 Z +Z +Z 3 (d) This cannot be a covariance matrix Suppose it is, then σ 3 9 > σ σ 33 6, which contradicts the Cauchy Schwartz inequality You can also verify this by showing that the matrix is not positive semidefinite For example, the determinant is Also one of the eigenvalues is negative (λ 08056) Alternatively, we can directly show that this matrix does not satisfy the definition of positive semidefiniteness by [ ] < Gaussian random vector Given a Gaussian random vector X N(µ,Σ), where µ (5) T and 0 Σ (a) Find the pdfs of i X, ii X +X 3, iii X +X +X 3, iv X 3 given (X,X ), and v (X,X 3 ) given X (b) What is P{X +X X 3 < 0}? Express your answer using the Q function (c) Find the joint pdf on Y AX, where A [ ] (a) i The marginal pdfs of a jointly Gaussian pdf are Gaussian Therefore X N(,) ii Since X and X 3 are independent (σ 3 0), the variance of the sum is the sum of the variances Also the sum of two jointly Gaussian random variables is also Gaussian Therefore X +X 3 N(7,3) 4

5 iii Since X +X +X 3 is a linear transformation of a Gaussian random vector, X +X +X 3 [ ] X X, X 3 it is a Gaussian random vector with mean and variance µ [ ] 5 9 and σ [ ] Thus X +X +X 3 N(9,) iv Since σ 3 0, X 3 and X are uncorrelated and hence independent since they are jointly Gaussian; similarly, since σ 3 0, X 3 and X are independent Therefore the conditional pdf of X 3 given (X,X ) is the same as the pdf of X 3, which is N(,9) v We use the general formula for the conditional Gaussian pdf: X {X x } N ( Σ Σ (x µ )+µ,σ Σ Σ Σ In the case of (X,X 3 ) X, Σ [ ], Σ [ ], Σ 0 [ ] Therefore the mean and variance of (X,X 3 ) given X x are [ ] [ ] [ µ (X,X 3 ) X x ] [ ] [ ] 5 x +4 +, 0 [ ] [ ] [ ] [ ] [ ] 4 0 [ ] Σ (X,X 3 ) X Thus X and X 3 are conditionally independent given X The conditional densities are X {X x } N(x +4,3) and X 3 {X x} N(,9) (b) LetY X +X X 3 Similarlyaspart(a)iii, X +X X 3 isalineartransformation of a Gaussian random vector, X +X X 3 [ ] X X, X 3 it is a Gaussian random vector with mean and variance µ [ ] 5 5 and σ [ ] Thus X +X X 3 N(5,), ie, Y N(5,) Thus { (Y 5) P{Y < 0} P < (0 5) } ( ) 5 Q ) 5

6 (c) In general, AX N(Aµ X, AΣ X A T ) For this problem, Thus Y N µ Y Aµ X Σ Y AΣ X A T [ ([ ] [ ]) 9 6, 6 [ ] 5 [ ] 9, ] [ ] Gaussian Markov chain Let X, Y, and Z be jointly Gaussian random variables with zero mean and unit variance, ie, E(X) E(Y) E(Z) 0 and E(X ) E(Y ) E(Z ) Let ρ X,Y denote the correlation coefficient between X and Y, and let ρ Y,Z denote the correlation coefficient between Y and Z Suppose that X and Z are conditionally independent given Y (a) Find ρ X,Z in terms of ρ X,Y and ρ Y,Z (b) Find the MMSE estimate of Z given (X,Y) and the corresponding MSE (a) From the definition of ρ X,Z, we have where, ρ X,Z Cov(X,Z) σ X σ Z, Cov(X,Z) E(XZ) E(X)E(Z) E(XZ) 0 E(XZ), σ X E(X ) E(X) 0, σ Y E(Y ) E(Y) 0 Thus, ρ X,Z E(XZ) Moreover, since X and Z are conditionally independent given Y, E(XZ) E(E(XZ Y)) E[E(X Y)E(Z Y)] Now E(X Y) can be easily calculated from the bivariate Gaussian conditional density Similarly, we have E(X Y) E(X)+ ρ X,Yσ X σ Y (Y E(Y)) ρ X,Y Y E(Z Y) ρ Y,Z Y 6

7 Therefore, combining the above, ρ X,Z E(XZ) E[E(X Y)E(Z Y)] E(ρ X,Y ρ Y,Z Y ) ρ X,Y ρ Y,Z E(Y ) ρ X,Y ρ Y,Z (b) X, Y and Z are jointly Gaussian random variables Thus, the minimum MSE estimate of Z given (X,Y) is linear [ ] ρx,y Σ (X,Y) T, ρ X,Y [ ] [ ] E(XZ) ρx,z Σ (X,Y) T Z, E(YZ) Σ Z(X,Y) T [ ρ X,Z ρ Y,Z ] ρ Y,Z Therefore, [ ] Ẑ Σ Z(X,Y) TΣ X (X,Y) T Y [ ] [ ] [ ] ρ ρ X,Z ρ X,Y X Y,Z ρ X,Y Y [ [ ] ρ ρ X,Z ρ X,Y Y,Z ρ ρ X,Y X,Y ρ X,Y [ 0 ρ X,Y ρ Y,Z +ρ Y,Z ] [ X Y ], ][ ] X Y where the last equality follows from the result of (a) Thus, Ẑ [ ] [ ] X 0 ρ Y,Z ρ Y Y,Z Y The corresponding MSE is MSE Σ Z Σ Z(X,Y) TΣ Σ (X,Y) T (X,Y) T Z [ ] [ ] [ ] ρ ρ X,Z ρ X,Y ρx,z Y,Z ρ X,Y ρ Y,Z [ [ ] ρ ρ X,Z ρ X,Y Y,Z ρ ρ X,Y X,Y [ ] [ ] ρ 0 ρ X,Z Y,Z ρ Y,Z ρ Y,Z ][ ρx,z ρ Y,Z ] 7

8 7 Prediction of an autoregressive process Let X be a random vector with zero mean and covariance matrix α α α n α α Σ X α α α n for α < X,X,,X n are observed, find the best linear MSE estimate (predictor) of X n Compute its MSE We have α n α n Σ X α n α α n α By defining Y [ ] T, X X n we have α n Σ Y, α n Therefore, and Σ YX [ α n α ] T, Σ XY [ α n α ], σ x ˆX n Σ XY Σ Y Y [ α n α ] α n α n h T Y (where h T Σ XY Σ Y ) [ 0 0 α ] Y (since h T Σ Y Σ XY ) αx n ; MSE σ x Σ XY Σ Y Σ YX h T Σ YX [ 0 0 α ] α α n α Y 8

9 8 Noise cancellation A classical problem in statistical signal processing involves estimating a weak signal (eg, the heart beat of a fetus) in the presence of a strong interference (the heart beat of its mother) by making two observations; one with the weak signal present and one without (by placing one microphone on the mother s belly and another close to her heart) The observations can then be combined to estimate the weak signal by cancelling out the interference The following is a simple version of this application LettheweaksignalX bearandomvariablewithmeanµandvariancep, andtheobservations be Y X +Z (Z being the strong interference), and Y Z +Z (Z is a measurement noise), where Z and Z are zero mean with variances N and N, respectively Assume that X, Z and Z are uncorrelated Find the best linear MSE estimate of X given Y and Y and its MSE Interprete the results ThisisavectorlinearMSEproblem SinceZ andz arezeromean,µ X µ Y µ and µ Y 0 We first normalize the random variables by subtracting off their means to get X X µ, and [ ] Y Y µ Now using the orthogonality principle we can find the best linear MSE estimate ˆX of X To do so we first find [ ] [ ] P +N N Σ Y P and Σ N N +N YX 0 Thus, ˆX Σ T YXΣ Y Y [ P 0 ] [ ] N +N N Y P(N +N )+N N N P +N P [ ] (N +N ) N Y P(N +N )+N N The best linear MSE estimate is ˆX ˆX +µ Thus, P ˆX ((N +N )(Y µ) N Y )+µ P(N +N )+N N (P((N +N )Y N Y ))+N N µ) P(N +N )+N N The MSE can be calculated by MSE σ X Σ T YXΣ Y Σ YX P Y P [ ] (N +N ) N P(N +N )+N N P (N +N ) P P(N +N )+N N PN N P(N +N )+N N [ ] P 0 9

10 TheequationfortheMSEmakesperfectsense First, notethatifn andn areheldconstant but P goes to infinity, the MSE tends to N N N +N Next, note that if both N and N go to infinity, the MSE goes to σx, ie, the estimate becomes worthless Finally, note that if either N or N goes to 0, the MSE also goes to 0 This is because the estimator will then use the measurement with zero noise variance and perfectly determine the signal X 0

11 Solutions to Additional Exercises Worst noise distribution Consider an additive noise channel Y X+Z, where the signal X N(0,P) and the noise Z has zero mean and variance N Assume X and Z are independent Find a distribution of Z that maximizes the minimum MSE of estimating X given Y, ie, the distribution of the worst noise Z that has the given mean and variance You need to justify your answer The worst noise has Gaussian distribution, ie Z N(0,N) To prove this statement, we show that the MSE corresponds to any other distribution of Z is less than or equal to the MSE of Gaussian noise, ie MSE NonG MSE G We know for any noise, MMSE estimation is no worse than linear MMSE estimation, so MSE NonG LMSE Linear MMSE estimate of X given Y is given by ˆX Cov(X,Y) σy (Y E(Y))+E(X) P P +N Y, LMSE σ X Cov (X,Y) σ Y P P P +N NP P +N Note that LMSE only depends on the second moment of X and Z So MSE corresponds to any distribution of Z is always upper bounded by the same LMSE, ie MSE NonG NP P+N When Z is Gaussian and independent of X, (X,Y) are joint Gaussian Then MSE G is equal to LMSE, ie MSE G NP P+N Hence, which shows the Gaussian noise is the worst MSE NonG NP P +N MSE G, Jointly Gaussian random variables Let X and Y be jointly Gaussian random variables with pdf f X,Y (x,y) π 3/4 e (4x /3+6y /3+8xy/3 8x 6y+6) (a) Find E(X), E(Y), Var(X), Var(Y), and Cov(X,Y) (b) Find the minimum MSE estimate of X given Y and its MSE (a) We can write the joint pdf for X and Y jointly Gaussian as ( exp f X,Y (x,y) [ a(x µ X ) +b(y µ Y ) +c(x µ X )(y µ Y ) πσ X σ Y ρ X,Y ]),

12 where a ( ρ, b X,Y )σ X ( ρ, c X,Y )σ Y ρ X,Y ( ρ X,Y )σ Xσ Y By inspection of the given f X,Y (x,y) we find that a 3, b 8 3, c 4 3, and we get three equations in three unknowns To find µ X and µ Y, we solve the equations and find that Finally ρ X,Y c ab, σx ( ρ, X,Y )a σy ( ρ X,Y )b 4 aµ X +cµ Y 4, bµ Y +cµ X 8, µ X, µ Y Cov(X,Y) ρ X,Y σ X σ Y 4 (b) X and Y are jointly Gaussian random variables Thus, the minimum MSE estimate of X given Y is linear E(X Y) Cov(X,Y) σy (Y µ Y )+µ X (Y )+ 3 Y MMSE E(Var(X Y)) ( ρ XY)σ X Markov chain Suppose X and X 3 are independent given X Show that f(x,x,x 3 ) f(x )f(x x )f(x 3 x ) f(x 3 )f(x x 3 )f(x x ) In other words, if X X X 3 forms a Markov Chain, then so does X 3 X X By definition of conditional independence, f(x,x 3 x ) f(x x )f(x 3 x )

13 Therefore, using the definition of conditional density, f(x 3 x,x ) f(x,x,x 3 ) f(x,x ) f(x,x 3 x )f(x ) f(x x )f(x ) f(x x )f(x 3 x ) f(x x ) f(x 3 x ) We are given that X and X 3 are independent given X Then f(x,x,x 3 ) f(x )f(x x )f(x 3 x,x ) f(x )f(x x )f(x 3 x ), In this case X X X 3 is said to form a Markov chain Similarly, f(x,x,x 3 ) f(x 3 )f(x x 3 )f(x x,x 3 ) f(x 3 )f(x x 3 )f(x x ), This shows that if X X X 3 is a Markov chain, then X 3 X X is also a Markov chain 4 Proof of Property 4 In Lecture Notes #6 it was stated that conditionals of a Gaussian random vector are Gaussian In this problem you will prove that fact If [ ] Y is a zero-mean GRV then X {Y y} N ( Σ X XY Σ Y y, σ X Σ XYΣ Y Σ YX) Justify each of the following steps of the proof (a) Let ˆX bethebestmselinearestimateofx giveny Then ˆX andx ˆX areindividually zero-mean Gaussians Find their variances (b) ˆX and X ˆX are independent (c) Now write X ˆX +(X ˆX) If Y y then X Σ XY Σ Y y + (X ˆX) (d) Now complete the proof Remark: This proof can be extended to vector X (a) Let ˆX be the best MSE linear estimate of X given Y In the MSE vector case section of Lecture Notes #6 it was shown that ˆX and X ˆX are individually zero-mean Gaussian random variables with variances Σ XY Σ Y Σ YX and σx Σ XYΣ Y Σ YX, respectively (b) The random variables ˆX and X ˆX are jointly Gaussian since they are obtained by a linear transformation of the GRV [Y X] T By orthogonality, ˆX and X ˆX are uncorrelated, so they are also independent By the same reasoning, X ˆX and Y are independent (c) Now write X ˆX +(X ˆX) Then given Y y since X ˆX is independent of Y X Σ XY Σ Y y+(x ˆX), (d) Thus X {Y y} is Gaussian with mean Σ XY Σ Y y and variance σ X Σ XYΣ Y Σ YX 3

14 5 Additive nonwhite Gaussian noise channel Let Y i X + Z i for i,,,n be n observations of a signal X N(0,P) The additive noise random variables Z,Z,,Z n are zero mean jointly Gaussian random variables that are independent of X and have correlation E(Z i Z j ) N i j for i,j n (a) Find the best MSE estimate of X given Y,Y,,Y n (b) Find the MSE of the estimate in part (a) Hint: the coefficients for the best estimate are of the form h T [a b b b b a] (a) The best estimate of X is of the form ˆX n h i Y i i We apply the orthogonality condition E(XY j ) E( ˆXY j ) for j n: P n h i E(Y i Y j ) i n h i E((X +Z i )(X +Z j )) i n h i (P +N i j ) i There are n equations with n unknowns: P P +N P +N/ P +N/ n P +N/ n P P +N/ P +N P +N/ n 3 P +N/ n P P +N/ n P +N/ n 3 P +N P +N/ P P +N/ n P +N/ n P +N/ P +N h h h n h n By the hint, there are only degrees of freedom given, a and b Solving this equation using the first rows of the matrix, we obtain h h P 3N +(n+)p h n h n 4

15 (b) The minimum mean square error is MSE E(X ˆX)X n P P h i Y i P ( i ) (n+)p 3N +(n+)p 3PN 3N +(n+)p 5