16.584: Random Vectors

Transcription

1 : Random Vectors Define X : (X 1, X 2,..X n ) T : n-dimensional Random Vector X 1 : X(t 1 ): May correspond to samples/measurements Generalize definition of PDF: F X (x) = P[X 1 x 1, X 2 x 2,...X n x n ] F X (x) = P[X x], F X ( ) = 1 F X ( ) = 0 pdf : f X (x) = n F X (x) x 1 x 2... x n Conditional PDF: F X B (x B) = P[X x B] = P[X x,b] P[B] Total Probability : F X (x) = n i=1 F X Bi (x B i )P[B i ]

2 2 Joint PDF of two Random Vectors: Let X : n dimensional and Y : m dimensional vectors F XY (x, y) = P[X x, Y y] Joint pdf : f XY (x, y) = n+m F XY (x,y) x 1.. x n y 1... y m

3 3 Transformation of Random Variables: (g i : Functionally Independent) y 1 = g 1 (x 1, x 2,..x n ) y 2 = g 2 (x 1, x 2,..x n ) y n = g n (x 1, x 2,..x n ) Under assumption that following inverse transformations exist: x 1 = φ 1 (y 1, y 2,..y n ) x 2 = φ 2 (y 1, y 2,..y n ) x n = φ n (y 1, y 2,..y n )

4 4 Define the Jacobian of Transformations: J = φ 1 φ 1 y n y φ n y 1.. φ n y n (1) f Y (y) = f X(x) J Note: J = J 1 J = g 1 = f X (x) J g 1 x n x g n x 1.. g n x n (2)

5 5 Expectation Vectors: µ X = [µ 1...µ n ] T µ i = x x i f X (x)dx Also in terms of marginal pdf: µ i = x i f Xi (x i )dx i Covariance Matrix: K K = E [ (X µ)(x µ) T ] Where k ij = E[(X i µ i )(X j µ j )] = k ji Also note: k ii = σ 2 i : Variance of X i. K : Symmetric real matrix (for X i, real ) Correlation Matrix R = E[XX T ] K = R µµ T R = K + µµ T

6 6 X and Y are Uncorrelated if: R XY = E[XY T ] = µ X µ Y T Leading To: K XY = 0 X and Y are Orthogonal if: R XY = E[XY T ] = 0 X and Y are Independent if: f XY (x, y) = f X (x) f Y (y) Note: Independence Implies Uncorrelated

7 7 Correlated Gaussian Random Variables Defined using the covariance matrix K pdf defined as : f X (x) = 1 (2π) n/2 K e [1 2 (x µ)t K 1 (x µ)] For independent RVS in X: K is a diagonal matrix K = n i=1 σ 2 i In general for non-independent X, K is non-diagonal symmetric matrix

8 8 Diagonalization of K Let the eigenvectors of K : φ 1,...φ n The eigenvalues : λ 1, λ 2..., λ n Such that Kφ i = λ i φ i Define matrix U whose columns contain φ i, i = 1, 2,..n Note: U T U = I and U T = U 1 : Unitary Matrices Assuming normalized eigenvectors : φ T φ = 1 Define the diagonal matrix Λ as containing the eigenvalues It can be shown: U 1 KU = Λ The transformation by U on K is distance preserving and effectively decorrelates the elements of X Whitening Transformation: Variances are identical

9 9 Some required properties of matrices: Matrix M is a Positive Semi-Definite Matrix if: The quadratic form : q(z) = z T Mz 0 For all vectors z. M is Positive Definite if q(z) > 0 for all z. Theorem: A real symmetric matrix M is Pos. Definite iff all λ i are positive. Therefore a positive definite covariance matrix K will have all positive eigenvalues and detk > 0.

10 10 Objective: Simultaneous Diagonalization of two covariance Matrices: Let P, Q two n n real symmetric matrices. Assume P is positive definite. Problem: Find the matrix V such that : V T PV = I AND V T QV = Λ = diag(λ 1,...λ n ) where λ i satisfy the generalized eigenvalue problem: Qv i = λ i Pv i λ i and v i : Generalized eigenvalues and eigenvectors

11 11 Proof: Let φ i and r i, i = 1, 2,, n be the eigenvectors and eigenvalues of P. Representing U as the matrix of eigenvectors and M : diag(r i ) U T P U = M Since P is pos. def. all r i are positive Let Z : diag(r 1/2 i ) exists and is real Consider the Whitening transformation: Z T [ U T PU ] Z = Z T M Z = I (Z T U T )P(UZ) = I (UZ) T P (UZ) = I Define Matrix V = UZ

12 12 Question:What Happens if Q is transformed using V = UZ? Consider similarity transformation: (UZ) T Q (UZ) = Z T U T QUZ = A Note that: A is Symmetric A T = ( Z T U T QUZ ) T = Z T U T Q T U Z = Z T U T Q U Z A T = A

13 13 Therefore there exists a similarity transformation on A that diagonalizes W T A W = Λ = diag(λ 1, λ 2,...λ n ) Where W T W = I Therefore: W T Z T U T Q U Z W = Λ Where the new transformation matrix is : WZU Examine effect of transformation WZU on P Does it retain the whitening effect? Z T [U T P U] Z = I W T Z T [U T P U] Z W = W T IW = I Transformation WZU on P retains whitening property

14 14 Matrix that achieves simultaneous diagonalization of P and Q is: V = UZW, where U is the matrix of eigenvectors of P Z = 1/ M, M is the diagonal matrix of eigenvalues of P W is matrix that diagonalizes A = (UZ) T Q(UZ) V T P V = I det [V T P V ] = V P V = I = 1 P V = [V T ] 1 Therefore V 0 and V 1 exists

15 15 Consider V T Q V = Λ Q V = [V T ] 1 Λ Q V = [PV ]Λ Q v i = λ i (Pv i ) P 1 Q v i = λ i v i System of Generalized eigenvalue equations v i are eigenvectors of P 1 Q Normalize v i such that V T PV = I v i T Pv i = 1

16 16 Summary: Find eigenvalues of P 1 Q Find correspondong (unnormalized) eigenvectors of P 1 Q : ˆv i Find constants K i such that v i = K i ˆv i Satisifies: v T i Pv i = 1 i = 1, 2..n