«Random Vectors» Lecture #2: Introduction Andreas Polydoros

Transcription

1 «Random Vectors» Lecture #2: Introduction Andreas Polydoros

2 Introduction Contents: Definitions: Correlation and Covariance matrix Linear transformations: Spectral shaping and factorization he whitening concept he Karhunen-Loeve expansion Binary Hypothesis testing

3 Introduction Definition-Correlation and Covariance matrix: Random vectors come about either by sampling one random 2 process X u, t at t, t2,, t or by observing a number of processes X u, t, X u, t,, X u, t at the same time. Essentially the two ways are equivalent mathematically. X u X u, m X, E { X( u) } X( u, ) mx

4 Introduction he autocorrelation function is: * RX E { X( u) X ( u) } X( u,) * * = E X ( u, ),, X ( u, ) X( u, ) RX (,) RX (, 2) RX (, ) RX ( 2,) RX ( 2, 2) RX ( 2, ) = RX (, ) RX (,2 ) RX (, )

5 Introduction he covariance matrix is: { * ( )( ) } { } K E X u m X u m X X X { } { } = E X u X u m E X u E X u m + m m * * * * X X X X K = R m m * X X X X

6 Linear transformations Suppose we are given a random vector X u and we construct another random vector Y u through the linear transformation n= = HX( u) Y u ym = hmnxn; m=,2,, M Question: What is the second-moment description of Y u?

7 Linear transformations m Y { Y( u )} { X( u )}, h h, E E = E { Y( u) } = = E{ Y( u, M) } hm h M E{ X( u, ) } m = Hm Y X In the above derivation we claimed that: E hmn X u n = h X u n n= n= { }? (, ) mn E (, ) In others words we assumed that expectation and summation (, 2 ) can be interchanged, but this holds only if R t, tgf is finite. X

8 Linear transformations For the autocorrelation function of Y u : Y * { } R = E Y u Y u { * ( HX ( u) )( HX ( u) ) } * * { HX ( u) X ( u) H } * * E { } = E = E = H X u X u H R = HR H Y X *

9 White vectors A useful concept is that of a white vector W u, which is a random vector with mean m W = 0, and covariance matrix: 2 σ σ 0 0 RW = KW = σ I = σ σ where is a constant and is the identity matrix. his means w i I W u that all components of are uncorrelated with each other, with zero mean and variance. σ 2

10 Spectral Shaping W u Problem: Given the white vector, can we find a linear transformation such that the resultant vector X u = HW u has given mean m and given covariance matrix K? X X W( u) = HW( u) X u

11 Spectral Shaping Since, m = 0, it follows that W m X = Hm = W 0 he covariance matrix of X u is: R X = = σ HR H herefore, spectral shaping is equivalent to the following: W HH * 2 * Given a correlation matrix RX, find an H such that RX = HH * 2 ote: σ can be absorbed in the given RX by creating a new given X. Other names for this problem are matrix factorization, square root of a matrix R

12 Spectral Shaping Definition: A complex (real) matrix is called Hermitian symmetric iff: A A = A * Definition: A complex (real) matrix is called Unitary (orthogonal) iff: A * AA = I

13 Spectral Shaping heorem: if K is Hermitian symmetric then there exists a unitary matrix E such that λ * λ 2 K = EΛE Λ = λ Ν with λ n; n=,2,,, the eigenvalues of K (not necessarily distinct). In other words: Hermitian symmetric matrices are always diagonalizable. heorem: A necessary and sufficient condition for such a to be nonnegative definite is that λn 0; n=, 2,,. K

14 Spectral Shaping heorem: Let K be Hermitian symmetric. hen for each distinct (simple) eigenvalue there corresponds an eigenvector which is orthogonal (orthonormal) to all others.o each eigenvalue of multiplicity k there correspond k linearly independent eigenvectors, which are orthogonal to all eigenvectors of the rest eigenvalues. hese k eigenvectors can be made orthogonal by application of the Gram-Schmidt procedure In summary, every Hermitian ( ) matrix has orthonormal eigenvectors { e n}, associated with its eigenvalues { λ n}. n= n= In fact, matrix E consists of these e `s as its columns, i.e., n [ 2 ] E= e e e

15 Spectral Shaping Returning to the factorization problem, we want to find an H * * such that R = HH. Writing R EΛE (since R is Hermitian) we have X X = X R X = EΛE * = EΛ Λ = EΛ 2 2 * 2 2 * * ( Λ ) ( 2)( 2 EΛ EΛ ) H E H * E * = ; Λ 2 λ λ 2 λ

16 Spectral Shaping We have arrived at a solution where H = EΛ 2 However this solution is not unique. o see this, take any unitary matrix U and observe that: X ( Λ )( Λ ) R = HH = E E * 2 2 = * 2 * 2 ( EΛ ) UU ( EΛ ) 2 2 ( EΛ U)( EΛ U) = another H * *

17 Spectral Shaping * Sometimes we take U = E and the resulting H is given as: 2 2 * H = EΛ U = EΛ E his matrix is often called the square root of From an applications viewpoint this factorization is useful in simulation, i.e., creating a random vector with desired correlation properties, starting from a random number generator. ote: if mx 0 then the appropriate linear transformation is: X = HW + m X where the factorization is done on K, not on R. X R X X

18 Spectral Shaping Example: he required covariance matrix is: K X 2 2 = he eigenvalues are found by solving the characteristic equation: det K λ I = 0; n=,2,3 { X n } λ = 0, λ = λ =

19 Spectral Shaping Solving for the corresponding eigenvectors we get: λ= 0 e= 3 3 λ2 = e2 = e3 = 2 3

20 Spectral Shaping herefore, we could choose the linear transformation: = = [ 2 3 ] = H EΛ e e e W( u,) X = HW = W( u,2) 0 0 W( u,3) otice that does not depend on W u,. X

21 Spectral Shaping In the above, we solved the problem of spectral shaping which is equivalent to a covariance matrix factorization. he solution was unconstrained, i.e., we imposed no restrictions on the nature of the linear transformation H ow assume that we impose the constraint of the linear transformation being causal.

22 Spectral Shaping Definition: A causal linear transformation is equivalent to H being lower triangular, i.e., the wanted linear transformation is h 0 0 X ( u, ) W ( u,) h2 h22 = + 0 X ( u, ) W ( u, ) h h 2 h n X( u, n) = hnl W( u, l) ; n=,2,, l= he problem can now be restated as Find a lower-triangular matrix H such that: ote: his factorization is called the Cholesky factorization of positive definite matrices KX m X = HH *

23 Spectral Shaping Example (real-valued covariance matrix): k k2 k h 0 0 h h2 h k2 k22 k2 h2 h h22 h2 = k k k h h h 0 0 h 2 2 k = h h =± k 2 k = h h h = k h in the same manner we can find the rest of. h ij

24 Properties - Spectral Resolution Assume a real covariance matrix K X. We can rewrite the factorization KX = EΛE as e e2 K X = [ λe λ2e2 λe] e or K X λnenen = K X n= his shows that can be decomposed (resolved) into a sum of matrices, each of the form ee with weight λ. Τhe set of eigenvectors n constitutes a basis for the n -dimensional vector space n { e } = n n

25 Properties - Spectral Resolution Every deterministic vector A can be expanded into a series n= where an = A, en = A en is the projection of basis vector e n A = ane A on the hus, vector A can be described in terms of its projections dcdc along the coordinates { a n } { e n} = n n

26 Properties - Spectral Resolution It is clear that we can create random vectors by choosing these projections as random variables An u, i.e., n= ote: If the eigenvectors have the form with in the n-th position, then A = An u e [ ] n { } e = 0,0,,0,,0,0,,0 n A u A = A ( u)

27 Properties - Directional Preference var Suppose we are given the covariance matrix of some vector X( u) and would like to project this vector on some unitlength vector b ( 2 bn = ). he projection is the inner n= product: Y u = X u, b = X u b Assuming that 0, the variance of Y u equals 2 2 { } σy b KX b m X = { } i.e., the variance of is a quadratic functional of the K X { } { } Y u = = E Y u = E Y u Y u = E b X u X u b = Y( u ) { b n } `s

28 Properties - Directional Preference Directional preference translates to finding those 2 directions b where the variance σ Y = b KXb is highest (or lowest). his is an optimization problem where we want to maximize the above quadratic form, subject to the unit-norm constraint. o solve this, we expand b on the orthonormal basis, i.e., { e n} = n b = bne n= 2 so that bn =. he quadratic form can now be written n= as: 2 σy = bkb X = be n n K X be m m = n= m= n n= m= bb e K e n m n X m

29 Properties - Directional Preference 2 Recalling that K e = λ e, can be written as X m m m 2 σy = bb n menλmem = λmbb n me nem ow the original problem can be equivalently stated as follows: { } { } Let u 2 n bn. We want to maximize U = λnun n subject to the constraint = un = and ui 0, λi 0 σ Y n= m= n= m= δnm σ 2 2 Y = λnbn n= n=

30 Properties - Directional Preference Example ( = 2): > 2 For λ2 λ the optimal solution is u = 0, u =. he general solution is to choose u m = where λm = max { λn} 2 and u n = 0 for n m. Since bi = ; i=, 2,,, it follows that: b =± m b = 0; n m n

31 Properties - Directional Preference he resulting variance is the maximum eigenvalue { } σ = λ = max λ 2 Y m n Recalling that b= bnen it follows that n= b max = where emax is the eigenvector of K X corresponding to the largest eigenvalue e max Question: What is the direction that minimizes the variance?

32 he whitening concept Converse to the factorization or spectral shaping problem Problem statement: Given a random vector X u with some mean m X and covariance K X, find a linear transformation G such that the output W( u) is a white vector m X = 0 X( u) W( u) m X 0 X( u) W( u) m X

33 he whitening concept From previous theory we know, that the covariance matrix of the output vector W u is For W u to be white we require K W = GK G We also know that K X can be factorized as (assuming real matrices) K = HH hus, we require the following equality to hold: GHH G = I X GH GH = I X KW = I

34 he whitening concept he simplest form of G that satisfies this equality is G= H 2 However, since H = EΛ U, we can express G in terms of E and Λ as 2 G= H = EΛ U = U 2 Recalling that U is by definition an arbitrary unitary matrix and E is also unitary since its columns are the orthonormal eigenvectors of, we end up at K X Λ E 2 G= U Λ Ε

35 he Karhunen-Loeve expansion Starting from the coloring problem equation, we define the following random vectors 2 = Λ X u E UW u Yu UW( u) 2 Λ Y( u) Z u Claim: Vector Y u = UW u is also white, and vector ghj Z ( u) = Λ gj 2 Y( u) has uncorrelated components, each with a different variance

36 he Karhunen-Loeve expansion Proof: Using the standard formulas we obtain my = UmX = 0 Y u KY = UKWU = UU = I is white m K Z Z 2 = Λ my = 0 ( Λ ) 2 2 = Λ KY 2 2 = λn = = Λ I Λ = Λ λ 0 = 0 λ { n} var Z u ; n, 2,,

37 he Karhunen-Loeve expansion Rewriting the coloring problem equation as we have: Z( u) X( u) = [ e e2 e ] Z ( u) X( u) = Zn( u) en n= λ X u = n= W u e n n n = EZ( u) X u his is the Karhunen Loeve expansion of X u. It states that every random vector can be written as a sum of orthonormal eigenvectors { e n }, each weighted by a random variable Wn ( u) and further scaled by λn

38 he Karhunen-Loeve expansion ote that the Karhunen Loeve expansion of a random vector X( u) is simply an expansion on a certain basis ({ e n }) of the -dimensional vector space. However, the basis is special, since (as we just showed) the projections X( u), en are uncorrelated random variables with variance λ n. (Projecting on an arbitrary basis, would not have the same effect) One could say that a random vector has preferences into how it is going to be distributed in space!