An Accurate Incremental Principal Component Analysis method with capacity of update and downdate

Size: px

Start display at page:

Download "An Accurate Incremental Principal Component Analysis method with capacity of update and downdate"

Adam Reed
6 years ago
Views:

1 0 International Conference on Computer Science and Information echnolog (ICCSI 0) IPCSI vol. 5 (0) (0) IACSI Press, Singapore DOI: /IPCSI.0.V5. An Accurate Incremental Principal Component Analsis method with capacit of update and downdate Wang Li, Chen Shuo and Wu Chengdong School of Information Science & Engineering, Northeastern Universit, Shenang, China Abstract Principal Component Analsis is a popular and powerful method in man machine learning tas, he traditional PCA is implemented in batch mode, which means the much lower efficienc, especiall for the tas which training dataset is updated or downdated frequentl, so it is reasonable to develop the incremental version of PCA. However, most of the existing incremental PCA is based on approximation with high estimation error, or lac of the downdate function. In this paper, a new accurate IPCA algorithm (AIPCA) which can provide both update and downdate capacit with higher accurac because of direct accurate algebraic derivation is proposed based on the matrix additive modification. Experimental analsis is also given for evaluating the time cost and calculation accurac of the AIPCA, the result demonstrates that the proposed method has high calculation accurac and acceptable time consuming. Kewords-Incremental PCA; update; downdate; SVD; Accurate. Introduction Principal Component Analsis (PCA) is one of the most useful techniques in multivariate analsis, and is emploed b man applications of pattern recognition and signal processing in last decades [] [] [3] [4]. However, the original PCA is performed in batch mode, which means all training data need to be included in the stage when calculation is carried on. When some new observations are incorporated in training dataset or some are removed out, the dataset s projection result of PCA has to be recalculated directl from all training data, his is a extremel time-consuming process. Furthermore, if the data are obtained from ran- r manifold which embedded in dimension- d observation space ( r d ), onl the form r principal components (PCs) are valuable for revealing the data pattern, this suggested that a new PCA algorithm can onl focus on the form r PCs calculation in this condition to achieve low time computational complexit but not all d PCs which is implemented in the original PCA. o overcome the disadvantage of time-consuming of the original PCA when update or downdate training dataset, some incremental versions are developed b researchers. he new PCs generated b the existing incremental PCA (IPCA) algorithms [5] [6] [7] are the estimation of batch mode PCA. he candid covariance-free IPCA (CCIPCA) [5] is one superior method of the recent wors, it gives a simple form and with higher estimation accurac and faster computing speed. But the CCIPCA s estimation accurac is limited b the number of samples, or in other words, the number of iterations. he PCs of CCIPCA will converge to batch mode result when new data are added in training set continuall, however it does not guarantee an acceptable estimation error upper bound between the two adjacent iterations. In some learning tas, training set is often changed frequentl, so the unguaranteed error ma led to unreasonable PCs which ma cause failure of the tas. In viewing of this limitation, a new IPCA called SVDU-IPCA [9] based on a singular value decomposition (SVD) updating algorithm [0] is proposed. he SVDU-IPCA have been mathematicall proved that the estimation error is bounded, but it onl provide the updating function, and cannot deal with the downdating situation when some data are removed from address: wl986_ren_ren@63.com 8

2 training set. Besides this deficienc, Matthew Brand [] also points out that the SVD updating algorithm emploed b SVDU-IPCA requires a full SVD of a dense matrix on each update, which reduces the performance. So there are strong reasons to design a new accurate IPCA algorithm which can calculate new PCs incrementall through direct accurate algebraic derivation. In this paper, we focus on the solution of a ind of PCA learning issue that is ver common in the field of image or video pattern recognition, in which sample dimension d is usuall larger than training samples number q. A new accurate IPCA algorithm (AIPCA) which can provide both update and downdate capacit is proposed in this paper based on the matrix additive modification presented in []. Experimental analsis is also given for evaluating the time cost and calculation accurac of the AIPCA.. Batch updating/downdating of SVD Incremental version of SVD has been studied in several literatures [0] [] [] [3], A derivation of batch updating/downdating algorithm of SVD proposed b Matthew Brand in [] is emploed in this paper for the theoretical basis of AIPCA. he capacit of update/downdate is obtained through matrix additive modification. p q Suppose that the intrinsic dimension of sample data is r, let data matrix Y have econom SVD rr Y U S V with S, so the update question can be translated into searching a incremental SVD solution of rr p l Y B U S V, where S, B. And the downdate question can be translated u u u into finding a incremental SVD solution of rr Sd, u Y U S V, where d d d Y is a submatrix of Y Y D, p l D. he procedure of the SVD update/downdate algorithm [] is described as follows:.. Update ) Obtain the QR decomposition Q R I U U B V 0( ql) l QR I V 0lr 0lr Il l ) Suppose the ran of R and R are r and r, obtain the matrixes the former r and r columns of Q and Q respectivel, and r rows of R and R respectivel. 3) Obtain the smaller sparse matrix R and Q and R which are formed b the former r Q which are formed b U B V K 0 I R l( ql) ll R 0lr S rr r r r r 4) Obtain the SVD decomposition K U S V S rr 5) Obtain the SVD decomposition of Y B 9

3 Y B U S V u u u V U QU S QV 0 lr.. Downdate ) Obtain the QR decomposition Q R U U I D Q R I V V 0 l I ll ( ql) ) he same as the second step of update procedure, obtain Q, 3) Obtain the smaller sparse matrix Q, R, R. U D K 0 R I V R S 0rr 0r r 0r r l( ql ) ll 4) he same as the forth step of update procedure, obtain U, 5) Obtain the SVD decomposition formed b the former q l rows of V Q V. 3. Accurate IPCA (AIPCA) S and V. U U Q U, S S, V d d is Y UdSdV d, where d here are three was to achieve PCA, which are eigendecomposition of data covariance matrix, SVD of data matrix and SVD of data inner product matrix respectivel. AIPCA focus on resolution of high dimension accurate IPCA issue that has the sample dimension d larger than training samples number q, which is ver common in the field of image or video pattern recognition, it means the size of data inner product matrix is smaller than the other two. So inspired b SVDU-IPCA, SVD of data inner product matrix is chosen to develop AIPCA. 3.. Update Let data matrix X [ x, x,, x q ] has q observations of dimension p. Suppose all data are sampled from a manifold, and the lowest dimension of linear space in which the data manifold can be embedded is r, A is new data matrix with c observations. he inner product matrix of X is XX, so the inner product matrix of new data matrix X A is X A X A. It is eas to prove that the inner product matrix is real smmetric positive semidefinite, so in ran reserving SVD decomposition of ran r, according to the derivation in [9], can be written as P P P Q P Q 3 0

4 where P P U U is the ran r SVD decomposition of, and the matrix P U is the rr PCA projection result,. Q can be obtained as Q U [9]. It can be see that P Irr U is ran r SVD of P, so using the SVD updating algorithm presented in section to get the decomposition of P Q is a straightforward approach, that is P Q U S V then we can get P P P P P P P Q P Q V S V UV where U VP, S P, V VP. Finall the PCA is achieved b the incremental SVD of inner product of new data matrix X A. 3.. Downdate q c In downdate procedure we redefine the notation A, let A is formed b the data which needed to be deleted from training sample matrix X, and X is the reserved data, then a new training sample matrix after shifting the columns which will be deleted to the right side is X ˆ X A. he column shift is a SVD-invariant transform to the data matrix, which means the SVD of XX ˆ ˆ and XX is equal, so the inner product of ˆX can be re-written in ran reserving SVD as X X ˆ ˆ P P 3 XX ˆ XX, ˆ then the SVD of P P Q where U V is the singular value decomposition of is I rr U, rr. So similar to update procedure, the SVD-downdating algorithm presented in section can be emploed to obtain the SVD of P, that is P U S V P P P hen the downdate PCA can be achieved as X X P P V S U P P P P P P V S V P P P U V U S V where U V VP, S. P 4. Experimental analsis he time complexit of computing a full SVD is O pq min p, q, if other operation for calculating PCA based on SVD are pulsed, the complexit would be more higher. Man powerful algorithm such as CCIPCA has been proposed to overcome this question, however most of them sacrifice accurac in exchange for efficienc, or just a single direction method lie SVDU-IPCA that can onl achieve a update tas. Suppose there is a training dataset with samples number q, observation dimension p and intrinsic dimension r, for

AIPCA, the focus is ept on solve a subspace learning question which q is smaller than p and r are much less than p in a incremental manner, this is exactl the most conditions which faced b pattern

5 AIPCA, the focus is ept on solve a subspace learning question which q is smaller than p and r are much less than p in a incremental manner, this is exactl the most conditions which faced b pattern recognition problem. If there are c samples needed to be added in or removed out, the QR decomposition of AIPCA would tae O p r c time, the SVD of sparse matrix taes O r c 3 time, and the rotation of the subspaces taes O p qr c time [], which can save much time than O pq min p, q because of q p and r p. Experiments have been implemented to evaluate the accurac and efficienc of AIPCA against CCIPCA and batch mode PCA. We generate a high dimension but low ran dataset which satisf high dimensional zero mean Gaussian distribution of with small additive noise randoml. he batch mode PCA is realized to obtain the standard subspace of PCA which is used to compare with AIPCA and CCIPCA. he inner product of component on principle axis between standard subspace and the testing algorithm s subspace (AIPCA s or CCIPCA s) is emploed to evaluate the correlation which can indicate the accurac of algorithm. he correlation represented b inner product of two algorithms is shown as Fig.. It can be seen that onl the correlation of the pair of the first PCs of CCIPCA is near, the rest PCs cannot obtain a acceptable accurac between two adjacent update iteration. However, for AIPCA there are onl little error generated b numerical error. he experimental result demonstrates that AIPCA has high accurac because of the direct accurate algebraic derivation. he time cost of batch mode PCA, AIPCA and CCIPCA when intrinsic of training set is growing is also given as Fig., we can see that the efficienc of AIPCA is lower than CCIPCA but beat batch mode PCA, this is acceptable at the premise of ensuring the higher accurac. Figure. (a) he correlation between batch mode PCA and CCIPCA (a), batch mode PCA and AIPCA (b). (b) Figure. he time cost of batch-mode PCA, AIPCA and CCIPCA 5. Conclusion

6 A new accurate IPCA algorithm (AIPCA) which can provide both update and downdate capacit is proposed in this paper, we focus on the solution of a ind of PCA learning issue that is ver common in the field of image or video pattern recognition, in which sample dimension d is usuall larger than training samples number q. based on the matrix additive modification presented. he experiment shows that the proposed method has high calculation accurac and acceptable efficienc. How to reduce the time cost is the focus of the future research. 6. Acnowledgment his wor is supported b the National Natural Science Foundation of China (Grant No ) and the Fundamental Research Funds for the Central Universities (No. N ). he authors would lie to than for these support. 7. References [] Bashi, R Bhavi, Multiscale PCA with Application to Multivariate Statistical Process Monitoring, AIChE Journal, vol. 44, pp , Jul 998. [] F Castells, P Laguna, Principal component analsis in ECG signal processing, Eurasip Journal on Advances in Signal Processing, vol. 007, 007.K. Elissa, itle of paper if nown, unpublished. [3] J Yang, D Zhang and A F Frangi, et al., wo-dimensional PCA: A new approach to appearance-based face representation and recognition, IEEE ransactions on Pattern Analsis and Machine Intelligence, vol. 6, pp. 3-37, Januar 004. [4] L Haiping, K N Plataniotis and A N Venetsanopoulos, MPCA: Multilinear principal component analsis of tensor objects, IEEE ransactions on Neural Networs, vol. 9, pp. 8-39, Januar 008. [5] D Socai, A Leonardis, Weighted and Robust Incremental Method for Subspace Learning, Proceedings Ninth IEEE International Conference on Computer Vision, vol., pp , 003. [6] W Juang, Z Yilu and H We-Shiuan, Candid Covariance-Free Incremental Principal Component Analsis, IEEE ransactions on Pattern Analsis and Machine Intelligence, vol. 5, pp , August 003. [7] L Yongmin, On incremental and robust subspace learning, Pattern Recognition, vol. 37, pp , Jul 004. [8] Y Wongsawat, Fast PCA via UV Decomposition and Application on EEG Analsis, 009 3st Annual International Conference of the IEEE Engineering in Medicine and Biolog Societ, pp , 009. [9] Z Haitao, Y Pong-Chi and J Kwo, A Novel Incremental Principal Component Analsis and Its Application for Face Recognition, IEEE ransactions on Sstems, Man and Cbernetics, Part B (Cbernetics), vol. 36, pp , August 006. [0] Z Honguan, H D Simon, On Updating Problems in Latent Semantic Indexing, SIAM Journal of Scientific Computing, vol., pp , September 999. [] M Brand, Fast low-ran modifications of the thin singular value decomposition, Linear Algebra and Its Applications, vol. 45, pp. 0-30, Ma 006. [] J C Wan, K J Houng and L J Gu, On Updating the singular value decomposition, 996 International Conference on Communication echnolog Proceedings, vol., pp , 996. [3] J R Bunch, C P Nielsen, Updating the Singular Value Decomposition, Numerische Mathemati, vol. 3, pp. - 9,

Covariance Tracking Algorithm on Bilateral Filtering under Lie Group Structure Yinghong Xie 1,2,a Chengdong Wu 1,b

Applied Mechanics and Materials Online: 014-0-06 ISSN: 166-748, Vols. 519-50, pp 684-688 doi:10.408/www.scientific.net/amm.519-50.684 014 Trans Tech Publications, Switzerland Covariance Tracking Algorithm