New Routes from Minimal Approximation Error to Principal Components

Transcription

1 New Routes from Miimal Approximatio Error to Pricipal Compoets Abhilash Alexader Mirada, Ya-Aël Le Borge, Gialuca Botempi Machie Learig Group, Départemet d Iformatique Uiversité Libre de Bruxelles, Boulevard du Triomphe - CP212 Brussels, Belgium 1050 September 12, 2007 Abstract. We itroduce two ew methods of derivig the classical PCA i the framework of miimizig the mea square error upo performig a lower-dimesioal approximatio of the data. These methods are based o two forms of the mea square error fuctio. Oe of the ovelties of the preseted methods is that the commoly employed process of subtractio of the mea of the data becomes part of the solutio of the optimizatio problem ad ot a pre-aalysis heuristic. We also derive the optimal basis ad the miimum error of approximatio i this framework ad demostrate the elegace of our solutio i compariso with a existig solutio i the framework. Keywords: pricipal compoets aalysis, eigevalue, matrix trace 1. Itroductio The problem of approximatig a give set of data usig a weighted liear combiatio of a fewer umber of vectors tha the origial dimesioality is classic. May applicatios that require such a dimesioality reductio desire that the ew represetatio retai the maximum variability i the data for further aalysis. A popular method that attais simultaeous dimesioality reductio, miimum mea square error of approximatio ad retaimet of maximum variace of the origial data represetatio i the ew represetatio is called the Pricipal Compoets Aalysis (PCA) (Hotellig, 1933; Jolliffe, 2002). The most popular framework for derivig PCA starts with the aalysis of variace. A very commo derivatio of PCA i this framework geerates the basis by iteratively fidig the orthogoal directios of maximum retaied variaces (Hotellig, 1933; Jolliffe, 2002; Mardia et al., 1979; Johso ad Wicher, 1992). Sice variace is implied i the statemet of the problem here, the mea is subtracted from the data as a prelimiary step. The secod most predomiat framework derives PCA by miimizig the mea square error of approximatio (Duda et abalexa@ulb.ac.be, Tel: , Fax: c 2007 Kluwer Academic Publishers. Prited i the Netherlads. NPLafterFeedbackv15.tex; 12/09/2007; 21:54; p.1

2 2 A. A. Mirada et al. al., 2001; Diamataras ad Kug, 1996; Bishop, 2006). Aided by the derivatio i the variace-based framework above, it has become acceptable to resort to mea subtractio of the data prior to ay aalysis i this framework too i order to keep the aalysis simple. I this letter our focus is o the latter framework withi which we demostrate two distict ad elegat aalytical methods of derivig the PCA. I each of these methods of derivatio, subtractio of data mea becomes part of the solutio istead of beig a iitial assumptio. The letter is orgaized as follows: i Sectio 2 we describe the motivatio behid the eed for yet aother derivatio of the classical PCA. I particular, we highlight the issue of mea ceterig i Sectio 2.1. The otatios are itroduced i Sectio 2.2 ad the PCA problem ad its iterpretatios are discussed i Sectio 3. After reviewig a existig solutio i Sectio 4, we make it evidet i Sectio 5 that our two methods are due to two forms of the optimizatio fuctio. The we itroduce our two methods of solvig the PCA problem i Sectios 6 ad 7 ad arrive at a simple commo form of the optimizatio fuctio i both the methods. This is aalyzed further i Sectio 8 where we show the relatio of the variace to the optimal basis i PCA as well as the miimum approximatio error attaied i PCA. I Sectio 8.3, we revisit the existig solutio i our framework of PCA itroduced i Sectio 4 ad equate it with our approach. 2. Motivatio There are may stadard textbooks of multivariate ad statistical aalysis (Jolliffe, 2002; Mardia et al., 1979; Johso ad Wicher, 1992) detailig PCA as a techique that seeks the best approximatio of a give set of data poits usig a liear combiatio of a set of vectors which retai maximum variace alog their directios. Sice this framework of PCA starts by fidig the covariaces, the mea has to be subtracted from the data ad becomes the de facto origi of the ew coordiate system. The subsequet aalysis is simple: fid the eigevector correspodig to the largest eigevalue of the covariace matrix as the first basis vector. The fid the secod basis vector o which the data compoets bear zero correlatio with the data compoets o the first basis vector. This turs out to be the eigevector correspodig to the secod largest eigevalue. I successively fidig the basis vectors that have ucorrelated compoets as the eigevectors of decreasig retaied variaces, the secod order cross momets NPLafterFeedbackv15.tex; 12/09/2007; 21:54; p.2

3 New Routes from Miimal Approximatio Error to Pricipal Compoets 3 betwee the compoets are successively elimiated 1. Computatioally, a widely employed trick i this framework fids the eigevectors usig sigular value decompositio of the mea cetered data matrix which effectively diagoalizes the covariace matrix without actually computig it (Jolliffe, 2002; Mardia et al., 1979). The set of orthogoal vectors correspodig to the largest few sigular values proportioal to the variaces yield those directios which retai the maximum variace i the ew represetatio of the data. The secod framework derives the PCA approximatio by usig its property of miimizig the mea square error. We thik that this framework is more effective i itroducig PCA to a ovice because the two outcomes of optimal dimesioality reductio, viz. error miimizatio ad retaied variace maximizatio, are attaied here simultaeously. Followig the path of the retaied variace maximizatio framework ad to keep the aalysis simple, may textbooks (Johso ad Wicher, 1992; Diamataras ad Kug, 1996; Hyvarie et al., 2001; Ripley, 1996) advocate a mea subtractio for this framework too without sesible justificatio. Pearso stated i his ow classical paper (Pearso, 1901): The secod momet of a system about a series of parallel lies is always least for the lie goig through the cetroid. Hece: The bestfittig straight lie for a system of poits i a space of ay order goes through the cetroid of the system. A procedure equivalet to rephrasig of this statemet is followed i a much refereced textbook (Duda et al., 2001) which reasos that sice the mea is the zero-dimesioal hyperplae which satisfies the miimum average square error criterio, ay higher dimesioal hyperplae should be excused to pass through it too. I order to keep our aalysis coheret with the cocept of simultaeous dimesioality reductio, retaied variace maximizatio ad approximatio error miimizatio, we do ot ivite the reader to such geometric ituitios. Note that the error miimizatio framework ca also be viewed as a total least squares regressio problem with all variables thought to be free so that the task is to fit a lower dimesioal hyperplae that miimizes the perpedicular distaces from the data poits to the hyperplae (Va Huffel, 1997). We will also be reviewig (Bishop, 2006) who derives PCA i the same framework as that of ours. Ulike i their approach, we either udertake a complete decompositio or force ay basis vectors to bear a commo statistic eticed by the prospect of a evetual mea 1 Elimiatio of higher order cross momets is dealt i Idepedet Compoets Aalysis (ICA) (Hyvarie et al., 2001). NPLafterFeedbackv15.tex; 12/09/2007; 21:54; p.3

4 4 A. A. Mirada et al. subtractio. Also for the beefit of practitioers who would like to deal data as realizatios of a radom variable, our treatmet i the data samples domai ca be readily exteded to a populatio domai To mea ceter or ot I the framework of fidig the basis of a lower dimesioal space which miimizes the mea square error of approximatio, the process of mea subtractio has so far bee part of the heuristics that the data eed to be cetered before istallig the ew low-dimesioal coordiate system motivated by the philosophy accordig to (Pearso, 1901) that, had the mea of the data ot bee subtracted, the best fittig hyperplae would pass through the origi ad ot through the cetroid. But there exist situatios where a hyperplae is merely expected to partitio the data space ito orthogoal subspaces ad as a result subtractio of mea is ot desired. Note that i such situatios, the term pricipal compoet does ot strictly hold as the basis vectors for the ew space are ot obtaied from the data covariace matrix ad the mai cocer there is the decompositio of the data rather tha approximatio. Oe such set of situatios are addressed by the Fukuaga-Kootz Trasform (Fukuaga ad Kootz, 1970; Mirada ad Whela, 2005) ad it works by ot requirig a subtractio of mea but istead fids the pricipal compoets of the autocorrelatio matrices of two classes of data. It is widely used i automatic target rejectio where eigevalue decompositio geerates basis for a target space orthogoal to the clutter space. But such is the issue of mea subtractio i usig this trasform that researchers of (Mahaalobis et al., 2004) ad (Xuo et al., 2003) use autocorrelatio ad covariace matrices, respectively, for the same task without a justificatio of the impact of their choice to mea ceter or ot. A similar approach called Eigespace Separatio Trasformatio (Plett et al., 1997) aimed at classificatio also does ot ivolve mea subtractio. A family of techiques called Orthogoal Subspace Projectio that is widely applied i oise rejectio of sigals use data that are ot mea cetered for the geeralized PCA that follows (Harsayi ad Chag, 1994). Although the theory of PCA demads mea subtractio for optimal low dimesioal approximatio, for may applicatios it is ot without cosequece. For example, the researchers of ecology ad climate studies have extesively debated the purpose ad result of mea ceterig for their PCA-based data aalysis. I (Noy-Meir, 1973), the characteristics ad apparet advatages of the pricipal compoets geerated without mea subtractio are compared for data sampled homogeously i the origial space or otherwise. The claim made therei is that if data NPLafterFeedbackv15.tex; 12/09/2007; 21:54; p.4

5 New Routes from Miimal Approximatio Error to Pricipal Compoets 5 form distict clusters, the ifluece of variace withi a cluster o aother ca be miimized by ot subtractig the mea. Aother ogoig debate amed Hockey Stick cotroversy (McItyre ad McKitrick, 2005) ivolves the appropriateess of mea subtractio for PCA i a much cited global warmig study (Ma et al., 1998). It should be bore i mid that this letter is either solely about the aforemetioed issue of mea ceterig that researchers usig PCA ofte take it for grated or does it chage the results of PCA that is previously kow to them. But we demostrate i a ew comprehesive framework that (i) the mea subtractio becomes a solutio to the optimizatio problem i PCA ad we reach this solutio through two simple distict methods that borrow little from traditioal textbook derivatios of PCA, ad (ii) the derivatio of the basis for the low dimesioal space coverges to miimum approximatio error ad maximum retaied variace i the framework. Cosequetly, we believe that may problems which raise questios about their choice regardig mea subtractio ca be revisited with ease usig our proposed PCA framework Notatios While esurig clarity to our aalysis i this letter, we have tried to maitai its brevity through appropriate otatios as summarized i the table below. NPLafterFeedbackv15.tex; 12/09/2007; 21:54; p.5

6 6 A. A. Mirada et al. J q : error fuctio q : ew dimesioality p : origial dimesioality : umber of samples x k R p ; k th data sample ˆx k R p ; approximatio of x k θ R p ; ew geeral origi x k = x k θ R p e i R p ; i th orthoormal basis vector of R p W = [e 1 e q] R p q B = I WW T R p p W = [e q+1 e p ] R p p q z k R q ; depedet o x k b R p q ; a costat Tr(A) : Trace of the matrix A rak(a) : Rak of the matrix A µ R p ; sample mea S R p p ; sample covariace matrix λ i : i th largest eigevalue of S r = rak(s) 3. Problem Defiitio i the Sample Domai Let x k R p,k =1,..., be a give set of data poits. Suppose we are iterested i orthoormal vectors e i R p,i =1,..., q p whose resultat of weighted liear combiatio ˆx k R p ca approximate x k with a miimum average (sample mea) square error or i other words miimize J q (ˆx k )= 1 x k ˆx k 2. (1) The problem stated above meas that we eed a approximatio x k ˆx k such that q ) ˆx k = (e T i x k e i (2) i=1 so that we attai the miimum for J q. This approximatio assumes that the origi of all orthoormal e i is the same as that of the coordiate NPLafterFeedbackv15.tex; 12/09/2007; 21:54; p.6

7 New Routes from Miimal Approximatio Error to Pricipal Compoets 7 system i which the data is defied. We reformulate the approximatio ˆx k = θ + q i=1 ( ) e T i (x k θ) e i (3) to assume that the ew represetatio usig basis vectors e i has a geeral origi θ R p ad ot the origi as i the approximatio (2). We assume orthoormality here because (i) orthogoality guaratees liearly idepedet e i so that they form a basis for R q (ii) ormalizig e i maitais otatioal simplicity i ot havig to divide the scalars e T i x k i (2) by the orm e i which is uity due to our assumptio. Hece, the PCA problem may be defied as argmi e i,θ 1 ( ) e T i (x k θ) e i ; ˆx k = θ + q x k ˆx k 2 i=1 : e T i e j =0,i j; e T i e i =1 i, j. which seeks a set of orthoormal basis vectors e i with a ew origi θ which miimizes the error fuctio i (1) i order to fid a lowdimesioal approximatio W T (x k θ) R q for ay x k R p, where It is ow easy to see that (3) becomes (4) W =[e 1 e q ]. (5) ˆx k = θ + WW T (x k θ). (6) Hece the displacemet vector directed from the approximatio ˆx k towards x k is x k ˆx k =(x k θ) WW T (x k θ), which usig x k = x k θ ca be writte cocisely as x k ˆx k = x k WW T x k. By settig B = I WW T for simplicity of otatio, we write the displacemet vector as x k ˆx k = B x k. (7) 4. Review of a existig solutio The PCA solutio i the framework of approximatio error miimizatio is derived i (Bishop, 2006) is reviewed here. They derive PCA by udertakig a complete decompositio ˆx k = Wz k + Wb (8) ito basis vectors cotaied i the colums of matrix W of (5) ad W =[e q+1 e p ] R p p q such that compoets of z k R q deped NPLafterFeedbackv15.tex; 12/09/2007; 21:54; p.7

8 8 A. A. Mirada et al. o x k, whereas compoets of b R p q are costats commo for all data poits. By takig derivative of the error fuctio with respect to b, they fid that b = W T µ (9) so that the commo compoets are those of the sample mea vector µ. This implies that by subtractig the sample mea they are o loger obliged to retai the p q dimesios correspodig to the colums of W which preserve little iformatio regardig the variatio i the data. The first drawback of this approach is that it couples the process of dimesioality reductio with mea subtractio although the two are show to be idepedet i our derivatio. By takig derivative of the error fuctio with respect to z k, they also show that z k = W T x k. Hece the approximatio they are seekig is ˆx k = WW T x k + W W T µ. (10) The secod drawback of their approach is the requiremet of yet aother costraied miimizatio of the error fuctio before they reach the solutio for the optimal colums of W. 5. Methods of PCA We have discussed the eed for a ew derivatio of PCA by (i) explaiig the lack of proper justificatio i the literature for subtractig the mea i a miimum mea square error framework, (ii) justifyig its chroic ecessity i may applicatios i Sectio 2, ad (iii) reviewig a recet attempt to solve this problem i Sectio 4. Our derivatios of the solutio for the problem i (4) are due to two simple forms of the error fuctio J q of (1) which we state as follow: Form 1 : J q (ˆx k )= 1 Form 2 : J q (ˆx k )= Tr (x k ˆx k ) T (x k ˆx k ) (11) ( 1 ) (x k ˆx k )(x k ˆx k ) T (12) We aalyze Form 1 i (11) i Sectio 6 to arrive at a simplified J q which is exactly the same as we get by followig a differet method of aalyzig Form 2 i (12) i Sectio 7. These two methods take differet paths towards the commo error fuctio, viz., the first usig straightforward expasio of the terms i J q ad the secod usig the property of matrix trace. The commo form of J q is subsequetly NPLafterFeedbackv15.tex; 12/09/2007; 21:54; p.8

9 New Routes from Miimal Approximatio Error to Pricipal Compoets 9 treated i Sectio 8 to reveal the rest of the solutio to our origial problem. 6. Aalysis of Form 1 of error fuctio Usig (7), the error fuctio J q of Form 1 i (11) ca be developed as J q (B, θ) = 1 x T k B T B x k. (13) The property that B = I WW T is idempotet ad symmetric, i.e., B = B 2 = B T, (14) or B is simply a orthogoal projector, may be used to reduce J q further as J q (B, θ) = 1 x T k B x k. (15) Expadig J q above usig x k = x k θ gives J q (B, θ) = 1 [ ] x T k Bx k 2θ T Bx k + θ T Bθ (16) I order to get the [ θ which miimizes J q, we fid the partial derivative J q / θ = 2B 1 ] x k θ ad settig it to zero results i θ = 1 x k = µ (17) which is as simple as regardig the sample mea of the data poits as the ew origi. Heceforth, we ca assume that x k is the data poit x k from which the sample mea has bee subtracted Simplifyig the error fuctio We may aalyze the error fuctio i (15) as follow: J q (W ) = 1 ( ) x T k I WW T x k = 1 x T k x k 1 x T k WW T x k = 1 ( [ ] ) 1 x T k x k Tr W T x k x T k W. NPLafterFeedbackv15.tex; 12/09/2007; 21:54; p.9

10 10 A. A. Mirada et al. We have the sample covariace matrix S = 1 x k x T k θ=µ (18) so that the term 1 x T k x k θ=µ equals Tr(S), ad we ca write ( ) J q (W )=Tr(S) Tr W T SW. (19) 7. Aalysis of Form 2 of error fuctio We ow aalyze the Form 2 of the error fuctio J q by substitutig (7) i (12) as ( [ ] ) 1 J q (B, θ) =Tr B x k x T k B T. (20) 7.1. Fidig θ As i the previous sectio, we deote the sample mea ad sample covariace matrix by µ ad S, respectively, ad we may develop the term i (20): 1 x k x T k = 1 = 1 (x k θ)(x k θ) T [x k x T k x k θ T θx T k + θθ T ] = S + µµ T µθ T θµ T + θθ T, (21) where we have used the sample autocorrelatio matrix (Fukuaga, 1990) 1 ( ( x k x T k =S + µµt. We get J q (B) = Tr B S + µµ T µθ T θµ T + θθ T ) B T ) upo substitutig (21) i (20). Usig (14) ad the cyclic permutatio property of trace of matrix products 2 we get ( ( )) J q (B) =Tr B S + µµ T µθ T θµ T + θθ T (22) ad usig the property of derivative of trace 3 ad the chai rule of derivatives 4, we fid that J q / θ =2B ( µ + θ) which whe equated 2 Tr (ΥΦΨ) = Tr (ΨΥΦ) = Tr (ΦΨΥ) 3 ( Tr ( ΨΦ T )) / Φ = Ψ 4 ( )/ u = [ ( )/ ( uv T )] v NPLafterFeedbackv15.tex; 12/09/2007; 21:54; p.10

11 New Routes from Miimal Approximatio Error to Pricipal Compoets 11 to zero results i leadig to the same solutio of Form 1 i (17) Simplifyig the error fuctio θ = µ (23) Havig foud θ, we ca substitute it i (22) to get J q (B) =Tr (BS). O ( substitutio ) for B i terms of W, we may write J q (W )=Tr (S) Tr WW T S. Utilizig the cyclic permutatio property of matrix trace agai, we get ( ) J q (W )=Tr (S) Tr W T SW. (24) 8. Optimal basis ad miimum error Note that we have arrived at the same set of equatios i both (19) ad (24) of Form 1 ad Form 2, respectively, whereby substitutig W as defied i (5) i either of them gives q J q (e i )=Tr(S) e T i Se i. (25) i= Relatio of variace to optimal basis Let us ow fid the variace λ i of the data projected o the basis vector e i. It is the average of the square of the differece betwee projectios e T i x k of the data poits ad the projectio e T i µ of the sample mea, i.e., λ i = 1 = 1 = e T i [ 1 ( ) 2 e T i x k e T i µ ( )( ) T e T i x k e T i µ e T i x k e T i µ ] (x k µ)(x k µ) T e i = e T i Se i. (26) Thus, the term q i=1 et i Se i i (25) gives the portio of the total variace Tr (S) retaied alog the directios of orthoormal e i. Hece, we are lookig for vectors e i of the form λ i = e T i (Se i), which is satisfied if NPLafterFeedbackv15.tex; 12/09/2007; 21:54; p.11

12 12 A. A. Mirada et al. Se i = λ i e i. Such a relatio implies (e i,λ i ) form a eige-pair of S. Note that sice there is o uique basis for ay otrivial vector space, ay basis that spas the q dimesioal space geerated by the eigevectors of S are solutios to e i too. I (25), sice argmi e i J q = argmax e i q e T i Se i, (27) i=1 the vectors e i have to be the eigevectors correspodig to the q largest ( pricipal ) eigevalues of S. This is the classical result of the PCA Relatio of variace to miimum approximatio error It follows from (26) that the term q i=1 et i Se i = q i=1 λ i of (25) is the sum of the q pricipal eigevalues of S; this is the maximum variace that could be retaied upo approximatio usig ay q basis vectors. Also, Tr (S) = r i=1 λ i,r= rak (S) is the total variace i the data. Substitutig these i J q i (25) gives the differece of the total variace ad the maximum retaied variace; the result is the miimum of the elimiated variace. Hece, for λ i λ j, j > i, the miimum mea square approximatio error ca be expressed as J q = r λ i q λ i = r λ i. i=1 }{{} i=1 }{{} i=q+1 }{{ } total variace retaied variace elimiated variace (28) 8.3. Compariso of the reviewed solutio with the preset work I order to compare the solutio of (Bishop, 2006) reviewed i Sectio 4, let us first write the approximatio i (6) as ˆx k = WW T x k + Bθ. We kow from (17) ad (23) that θ = µ ad, hece, ˆx k = WW T x k + Bµ. (29) If W W T = B, we have the approximatio accordig to (Bishop, 2006) i (10) of Sectio 4 equivalet to the approximatio i (29). While the drawbacks of (6) highlighted i Sectio 4 exist, let us outlie the differece i these two approaches: we have demostrated i our solutios that the ew origi θ R p of the low dimesioal coordiate system should be the mea µ R p so that the error of the approximatio is reduced. But (Bishop, 2006) ecessitates a orthogoal projectio of certai data-idepedet compoets b R p q NPLafterFeedbackv15.tex; 12/09/2007; 21:54; p.12

13 New Routes from Miimal Approximatio Error to Pricipal Compoets 13 to µ R p to achieve the same objective. Our approach has show that such a dimesioality reductio coupled with mea subtractio is uecessary for derivig PCA Populatio PCA For populatio PCA (Mardia et al., 1979; Johso ad Wicher, 1992), where the samples that form the data are assumed to be realizatios of a radom variable, we have made it easy for the reader to follow 1 our aalysis by just replacig all occurreces of E, the expectatio operator; ad bold faces for radom variables as i x k x, ˆx k ˆx, ad x k x. 9. Coclusio Motivated by the eed to justify the heuristics of pre-aalysis mea ceterig i PCA ad related questios, we have demostrated through two distict methods that the mea subtractio becomes part of the solutio of the stadard PCA problem i a approximatio error miimizatio framework. We based these two methods o two subtly differet forms of the error fuctio. We have also derived the optimal basis ad the miimum error of approximatio i this framework ad have compared our results with a existig solutio. Ackowledgemets This work was fuded by the Project TANIA (WALEO II) of the Walloo Regio, Belgium. The authors thak their colleague Olivier Caele for his appreciable commets. Thaks are also due to Dr. P.P. Mohalal of ISRO Iertial Systems Uit, Idia for his valuable isights. The authors are very grateful to the editor-i-chief ad three aoymous reviewers for their excellet suggestios o a earlier versio of this letter. Refereces C. M. Bishop. Patter Recogitio ad Machie Learig. Iformatio Sciece ad Statistics. Spriger, New York, August K. I. Diamataras ad S. Y. Kug. Pricipal Compoet Neural Networks: Theory ad Applicatios. NewYork, February, R. O. Duda, P. E. Hart, ad D. G. Stork. Patter Classificatio. Wiley Itersciece, New York, secod editio, NPLafterFeedbackv15.tex; 12/09/2007; 21:54; p.13

14 14 A. A. Mirada et al. K. Fukuaga. Itroductio to Statistical Patter Recogitio. Computer Sciece ad Scietific Computig. Academic Press, Sa Diego, secod editio, K. Fukuaga ad W.L.G. Kootz. Applicatio of the Karhue-Loeve expasio to feature selectio ad orderig. IEEE Trasactios o Computers, C-19(4): , J. C. Harsayi ad C.-I. Chag. Hyperspectral Image Classificatio ad Dimesioality Reductio: A Orthogoal Subspace Projectio Approach. IEEE Trasactios o Geosciece ad Remote Sesig, 32(4): , July, H. Hotellig. Aalysis of a complex of statistical variables ito pricipal compoets. Joural of Educatioal Psychology, 24: , X. Huo, M. Elad, A. G. Flesia, B. Muise, R. Stafill, A. Mahalaobis, et al. Optimal Reduced-Rak Quadratic Classifiers usig the Fukuaga-Kootz Trasform with Applicatios to Automated Target Recogitio, Proceedigs of SPIE, 5094:59-72, September, A. Hyvarie, J. Karhue, ad E. Oja. Idepedet Compoet Aalysis, volume 27 of Adaptive ad Learig Systems for Sigal Processig, Commuicatios ad Cotrol. Wiley-Itersciece, New York, Jue, R. A. Johso ad D. W. Wicher. Applied Multivariate Statistical Aalysis. Pretice-Hall, Ic., Upper Saddle River, New Jersey, third editio, I. T. Jolliffe. Pricipal Compoet Aalysis. Spriger, 2d editio, New York, A. Mahaalobis, R. R. Muise, S. R. Stafill ad A. Va Nevel. Desig ad Applicatio of Quadratic Correlatio Filters for Target Detectio. IEEE Trasactio o Aerospace ad Electroic Systems, 40(3): , July, M. E. Ma, R. S. Bradley ad M. K. Hughes. Global-scale Temperature Patters ad Climate Forcig over the Past Six Ceturies. Nature, 392: , April, K. Mardia, J. Ket, ad J. Bibby. Multivariate Aalysis. Academic Press, Lodo, S. McItyre ad R. McKitrick. Reply to commet by Huybers o Hockey sticks, Pricipal Compoets, ad Spurious Sigificace. Geophysical Research Letters, 32, L20713, A. A. Mirada ad P. F. Whela. Fukuaga-Kootz Trasform for Small Sample Size Problems. Proceedigs of the IEE Irish Sigals ad Systems Coferece, pp , Dubli, I. Noy-Meir. Data Trasformatios i Ecological Ordiatio: I. Some Advatages of No-Ceterig. The Joural of Ecology, 61(2): , July, K. Pearso. O lies ad plaes of closest fit to systems of poits i space. Philosophical Magazie, 2: , G. L. Plett, T. Doi, ad D. Torrieri. Mie Detectio usig Scatterig Parameters ad a Artificial Neural Network. IEEE Trasactios o Neural Networks, 8(6): , November, B. D. Ripley. Patter Recogitio ad Neural Networks. Cambridge Uiversity Press, Cambridge, S. Va Huffel (Ed.), Recet Advaces i Total Least Squares Techiques ad Errorsi-Variables Modelig, SIAM, Philadelphia, PA, NPLafterFeedbackv15.tex; 12/09/2007; 21:54; p.14