Prncpal Component Transform Multvarate Random Sgnals A real tme sgnal x(t can be consdered as a random process and ts samples x m (m =0; ;N, 1 a random vector: The mean vector of X s X =[x0; ;x N,1] T M X = E(X =[E(x0; ;E(x N,1] T =[0; ; N,1] T The covarance matrx of X s X = E[(X, M X (X, M X T ]=E(XX T, MM T = :: j :: where j = E(x x j, j s the covarance of two random varables x and x j. When = j, j becomes the varance of x, = E(x,. The correlaton matrx of X s R X = E(XX T = :: r j :: where r j = j + j. Both X and R X are symmetrc matrces (Hermtan f X s complex. A sgnal vector X can always be easly converted nto a zero-mean vector X 0 = X, M X wth all of ts nformaton (or dynamc energy conserved. In the followng, wthout loss of generalty, we wll assume M X = 0 and therefore X = R X. 7 7 1
The Prncpal Component Transform The Prncpal Component Transform s also called Karhunen-Loeve Transform (KLT, Hotellng Transform, oregenvector Transform. Let and be the th egenvector and egenvalue of the correlaton matrx R X : R X = ( =0; ;N, 1 We can construct an N N matrx =[0; ; N,1] Snce the columns of are the egenvectors of a symmetrc (Hermtan f X s complex matrx R X,s orthogonal (untary: T =I.e., and we have,1 = T R X = where =dag(0; ; N,1. Or, we have,1 R X = T R X = We can now dene the orthogonal (untary f X s complex Prncpal Component Transform of X by ( Y = T X X =Y The th component of the forward transform Y = T X s the projecton of X on : y =( ;X= T X and the nverse transform X = Y represents X n the N-dmensonal space spanned by ( =0; 1; ;N, 1: N,1 X X = y
KLT Completely Decorrelates the Sgnal KLT s the optmal orthogonal transform n the followng sense: KLT completely decorrelates the sgnal KLT optmally compacts the energy (nformaton contaned n the sgnal. The rst property s smply due to the denton of KLT, and the second property s due to the fact that KLT redstrbutes the energy among the N components n such a way that most of the energy s contaned n a small number of components of Y = T X. To see the rst property, consder the correlaton matrx R Y of Y : R Y = E(YY T =E[ T X( T X T ] = E[ T (XX T ]= T E(XX T = T R X = We see that after KLT, the correlaton matrx of the sgnal s dagonalzed,.e., the correlaton r j =0between any two components x and x j s always zero. In other words, the sgnal s completely decorrelated.
KLT Optmally Compacts the Energy Consder a general orthogonal transform par dened as ( Y = A T X X = AY where X and Y arenby1vectors and A s an arbtrary N by N orthogonal matrx A,1 = A T. We represent A by ts column vectors A ; ( =0; ;N, 1 as or A =[A0; ;A N,1] A T = A T 0 : : A T N,1 Now the th component of Y can be wrtten as 7 y = A T X As we assume the mean vector of X s zero M X = 0 (and obvously we also have M Y = A T M x = 0, we have X = R X, and the varance of the th element nbothx and Y are x = E(x = E(e x and y = E(y = E(e y where e x = x and e y = y represent the energy contaned n the th component of X and Y, respectvely. In order words, the trace of X (the sum of all the dagonal elements of the matrx represents the expectaton of the total amount of energy contaned n the sgnal X N,1 X Total energy contaned n X = tr X = N,1 X x = E(x N,1 X =E( e x
Snce an orthogonal transform A does not change the length of a vector X,.e., k Y k=k AX k=k X k, where vu u X vu u X k X k= t N,1 x = t N,1 the total energy contaned n the sgnal vector X s conserved after the orthogonal transform. (Ths concluson can also be obtaned from the fact that orthogonal transforms do not change the trace of a matrx. We next dene S m (A = E(y = y = E(e y where m N. S m (A s a functon of the transform matrx A and represents the amount of energy contaned n the rst m components of Y = A T X. Snce the total energy s conserved, S m (A also represents the percentage of energy contaned n the rst m components. In the followng we wll show that S m (A s maxmzed f and only f the transform A s the KLT: S m (A = S m ( S m (A.e., KLT optmally compacts energy nto a few components of the sgnal. Consder = = e x E(y = E[A T X(AT XT ] E[A T X(X T A ] = A T E(XXT A A T R XA (1 Now we need to nd a transform matrx A so that ( Sm (A! max subject to A T j A j =1 (j =0; ;m, 1 The constrant A T j A j = 1 s to guarantee that the column vectors n A are normalzed. Ths constraned optmzaton problem can be solved by Lagrange multpler method as shown below.
We let @ @A [S m (A, j (A T j A j, 1] = 0 j=0 = @ [ (A T j @A R XA j, j A T j A j + j ] j=0 = @ [A T @A R XA, A T A ] = R x A, A =0 (* the last equal sgn s due to explanaton n the handout of revew of lnear algebra. We see that the column vectors of A must be the egenvectors of R X : R X A = A ( =0; ;m, 1.e., the transform matrx must be A =[A0; ;A N,1] ==[0; ; N,1] Thus we have proved that the optmal transform s ndeed KLT, and S m ( = T R X = where the th egenvalue of R X s also the average (expectaton energy contaned n the th component of the sgnal. If we choose those 0 s that correspond to the m largest egenvalues of R X : 0 1 m N,1, then S m ( wll acheve maxmum.