Uncorrelated Multilinear Principal Component Analysis through Successive Variance Maximization

Size: px

Start display at page:

Download "Uncorrelated Multilinear Principal Component Analysis through Successive Variance Maximization"

Brandon Adams
5 years ago
Views:

1 Uncorrelated Multilinear Principal Component Analysis through Successive Variance Maximization Haiping Lu 1 K. N. Plataniotis 1 A. N. Venetsanopoulos 1,2 1 Department of Electrical & Computer Engineering, University of Toronto 2 Ryerson University The 25th International Conference on Machine Learning ICML 2008 Haiping Lu, K. N. Plataniotis, A. N. Venetsanopoulos (U of T) Uncorrelated Multilinear PCA ICML / 27

2 Outline 1 Motivation 2 The Proposed UMPCA Algorithm Tensor-to-Vector Projection Uncorrelated Multilinear PCA (UMPCA) 3 Experimental Evaluations Experimental Setup Experimental Results 4 Conclusions Haiping Lu, K. N. Plataniotis, A. N. Venetsanopoulos (U of T) Uncorrelated Multilinear PCA ICML / 27

3 Tensorial Data. Tensor: multidimensional array. Generalization of vector (first-order) and matrix (second-order). N modes Nth-order tensor. Wide range of applications: images, video sequences, streaming and mining data. Haiping Lu, K. N. Plataniotis, A. N. Venetsanopoulos (U of T) Uncorrelated Multilinear PCA ICML / 27

4 Dimensionality Reduction Problem. Tensor objects are usually high-dimensional curse of dimensionality. Computationally expensive to handle. Many classifiers perform poorly in high-dimensional space given a small number of training samples. A class of tensor objects are mostly highly constrained to a subspace, a manifold of intrinsically low dimension. Dimensionality reduction (feature extraction): transformation to low-dimensional space while retaining most of the underlying structure. Haiping Lu, K. N. Plataniotis, A. N. Venetsanopoulos (U of T) Uncorrelated Multilinear PCA ICML / 27

5 Dimensionality Reduction Problem. Tensor objects are usually high-dimensional curse of dimensionality. Computationally expensive to handle. Many classifiers perform poorly in high-dimensional space given a small number of training samples. A class of tensor objects are mostly highly constrained to a subspace, a manifold of intrinsically low dimension. Dimensionality reduction (feature extraction): transformation to low-dimensional space while retaining most of the underlying structure. Haiping Lu, K. N. Plataniotis, A. N. Venetsanopoulos (U of T) Uncorrelated Multilinear PCA ICML / 27

6 Focus: Unsupervised Dimensionality Reduction-PCA. Linear method: principal component analysis (PCA) Produce uncorrelated features. Retain as much as possible the variations. Reshape tensors into vectors: high computational and memory demand, break natural structure in original data. Multilinear methods: feature extraction directly from tensors Tensor rank-one decomposition (TROD) [Shashua & Levin, 2001]. Two-dimensional PCA (2DPCA) [Yang et al., 2004]. Generalized low rank approximation of matrices (GLRAM) [Ye, 2005] & Generalized PCA (GPCA) [Ye et al., 2004]. Concurrent subspaces analysis (CSA) [Xu et al., 2005)]. Multilinear PCA (MPCA) [Lu et al., 2008)]. Haiping Lu, K. N. Plataniotis, A. N. Venetsanopoulos (U of T) Uncorrelated Multilinear PCA ICML / 27

7 Focus: Unsupervised Dimensionality Reduction-PCA. Linear method: principal component analysis (PCA) Produce uncorrelated features. Retain as much as possible the variations. Reshape tensors into vectors: high computational and memory demand, break natural structure in original data. Multilinear methods: feature extraction directly from tensors Tensor rank-one decomposition (TROD) [Shashua & Levin, 2001]. Two-dimensional PCA (2DPCA) [Yang et al., 2004]. Generalized low rank approximation of matrices (GLRAM) [Ye, 2005] & Generalized PCA (GPCA) [Ye et al., 2004]. Concurrent subspaces analysis (CSA) [Xu et al., 2005)]. Multilinear PCA (MPCA) [Lu et al., 2008)]. Haiping Lu, K. N. Plataniotis, A. N. Venetsanopoulos (U of T) Uncorrelated Multilinear PCA ICML / 27

8 Question to be Answered: Uncorrelated features Result in minimum redundancy. Ensure linear independence among features. Simplify classification task. Question: Can we extract uncorrelated features directly from tensor objects in an unsupervised way? Haiping Lu, K. N. Plataniotis, A. N. Venetsanopoulos (U of T) Uncorrelated Multilinear PCA ICML / 27

9 Outline 1 Motivation 2 The Proposed UMPCA Algorithm Tensor-to-Vector Projection Uncorrelated Multilinear PCA (UMPCA) 3 Experimental Evaluations Experimental Setup Experimental Results 4 Conclusions Haiping Lu, K. N. Plataniotis, A. N. Venetsanopoulos (U of T) Uncorrelated Multilinear PCA ICML / 27

10 Notations. Vector: lowercase boldface x. Matrix: uppercase boldface U. Tensor: calligraphic letter A. Subscript p: the feature index. Subscript m: the training sample index. Superscript (n): the n-mode. Superscript T : the transpose. Operation n : the n-mode multiplication. Haiping Lu, K. N. Plataniotis, A. N. Venetsanopoulos (U of T) Uncorrelated Multilinear PCA ICML / 27

11 Elementary Multilinear Projection (EMP). y = X 1 u (1)T 2 u (2)T... N u (N)T Haiping Lu, K. N. Plataniotis, A. N. Venetsanopoulos (U of T) Uncorrelated Multilinear PCA ICML / 27

12 Tensor-to-Vector Projection (TVP). y = X N n=1 {u(n)t p, n = 1,..., N} P p=1 Haiping Lu, K. N. Plataniotis, A. N. Venetsanopoulos (U of T) Uncorrelated Multilinear PCA ICML / 27

13 Outline 1 Motivation 2 The Proposed UMPCA Algorithm Tensor-to-Vector Projection Uncorrelated Multilinear PCA (UMPCA) 3 Experimental Evaluations Experimental Setup Experimental Results 4 Conclusions Haiping Lu, K. N. Plataniotis, A. N. Venetsanopoulos (U of T) Uncorrelated Multilinear PCA ICML / 27

14 The UMPCA Problem Formulation. Input Tensorial training samples {X m R I 1... I N, m = 1,..., M}. The desired subspace dimensionality P. Objective {u (n)t p, n = 1,..., N} = arg max S y T p, S y T p = M m=1 (y m p y p ) 2. s.t. u (n)t p = 1 and g p : the pth coordinate vector, u (n) p g T p gq g p g q = δ pq, p, q = 1,..., P. g p (m) = y mp = X m N n=1 {u(n)t p, n = 1,..., N}. Output The TVP {u (n)t p, n = 1,..., N} P p=1 satisfying the objective criterion. Haiping Lu, K. N. Plataniotis, A. N. Venetsanopoulos (U of T) Uncorrelated Multilinear PCA ICML / 27

15 The UMPCA Problem Formulation. Input Tensorial training samples {X m R I 1... I N, m = 1,..., M}. The desired subspace dimensionality P. Objective {u (n)t p, n = 1,..., N} = arg max S y T p, S y T p = M m=1 (y m p y p ) 2. s.t. u (n)t p = 1 and g p : the pth coordinate vector, u (n) p g T p gq g p g q = δ pq, p, q = 1,..., P. g p (m) = y mp = X m N n=1 {u(n)t p, n = 1,..., N}. Output The TVP {u (n)t p, n = 1,..., N} P p=1 satisfying the objective criterion. Haiping Lu, K. N. Plataniotis, A. N. Venetsanopoulos (U of T) Uncorrelated Multilinear PCA ICML / 27

16 The UMPCA Problem Formulation. Input Tensorial training samples {X m R I 1... I N, m = 1,..., M}. The desired subspace dimensionality P. Objective {u (n)t p, n = 1,..., N} = arg max S y T p, S y T p = M m=1 (y m p y p ) 2. s.t. u (n)t p = 1 and g p : the pth coordinate vector, u (n) p g T p gq g p g q = δ pq, p, q = 1,..., P. g p (m) = y mp = X m N n=1 {u(n)t p, n = 1,..., N}. Output The TVP {u (n)t p, n = 1,..., N} P p=1 satisfying the objective criterion. Haiping Lu, K. N. Plataniotis, A. N. Venetsanopoulos (U of T) Uncorrelated Multilinear PCA ICML / 27

17 The UMPCA Problem Formulation. Input Tensorial training samples {X m R I 1... I N, m = 1,..., M}. The desired subspace dimensionality P. Objective {u (n)t p, n = 1,..., N} = arg max S y T p, S y T p = M m=1 (y m p y p ) 2. s.t. u (n)t p = 1 and g p : the pth coordinate vector, u (n) p g T p gq g p g q = δ pq, p, q = 1,..., P. g p (m) = y mp = X m N n=1 {u(n)t p, n = 1,..., N}. Output The TVP {u (n)t p, n = 1,..., N} P p=1 satisfying the objective criterion. Haiping Lu, K. N. Plataniotis, A. N. Venetsanopoulos (U of T) Uncorrelated Multilinear PCA ICML / 27

18 The UMPCA Problem Formulation. Input Tensorial training samples {X m R I 1... I N, m = 1,..., M}. The desired subspace dimensionality P. Objective {u (n)t p, n = 1,..., N} = arg max S y T p, S y T p = M m=1 (y m p y p ) 2. s.t. u (n)t p = 1 and g p : the pth coordinate vector, u (n) p g T p gq g p g q = δ pq, p, q = 1,..., P. g p (m) = y mp = X m N n=1 {u(n)t p, n = 1,..., N}. Output The TVP {u (n)t p, n = 1,..., N} P p=1 satisfying the objective criterion. Haiping Lu, K. N. Plataniotis, A. N. Venetsanopoulos (U of T) Uncorrelated Multilinear PCA ICML / 27

19 The Approach of Successive Maximization. 1 Determine the first EMP {u (n)t 1, n = 1,..., N} by maximizing S y T 1 without any constraint. 2 Determine the second EMP {u (n)t 2, n = 1,..., N} by maximizing S y T 2 subject to the constraint that g T 2 g 1 = 0. 3 Determine the third EMP {u (n)t 3, n = 1,..., N} by maximizing S y T 3 subject to the constraint that g T 3 g 1 = 0 and g T 3 g 2 = Haiping Lu, K. N. Plataniotis, A. N. Venetsanopoulos (U of T) Uncorrelated Multilinear PCA ICML / 27

20 The Alternating Projection Method. The need for an iterative solution Simultaneous determination of N sets is infeasible. Alternating projection: solve one set with all the other sets fixed and iterate (e.g., Alternating Least Square). The alternating projection method 1 Assume that {u (n) p, n n } is given. 2 Project {X m } in these N 1 modes to {ỹ (n ) m p }. 3 Determine u (n ) p that projects {ỹ (n ) m p } onto a line with variance maximized, subject to zero-correlation. PCA with input {ỹ (n ) m p } and total scatter matrix S (n ) T p. Haiping Lu, K. N. Plataniotis, A. N. Venetsanopoulos (U of T) Uncorrelated Multilinear PCA ICML / 27

21 The Alternating Projection Method. The need for an iterative solution Simultaneous determination of N sets is infeasible. Alternating projection: solve one set with all the other sets fixed and iterate (e.g., Alternating Least Square). The alternating projection method 1 Assume that {u (n) p, n n } is given. 2 Project {X m } in these N 1 modes to {ỹ (n ) m p }. 3 Determine u (n ) p that projects {ỹ (n ) m p } onto a line with variance maximized, subject to zero-correlation. PCA with input {ỹ (n ) m p } and total scatter matrix S (n ) T p. Haiping Lu, K. N. Plataniotis, A. N. Venetsanopoulos (U of T) Uncorrelated Multilinear PCA ICML / 27

22 The UMPCA Solution. For p = 1 u (n ) 1 : the unit eigenvector of S (n ) T 1 associated with the largest eigenvalue. For p > 1 1 Let Ỹ(n ) p = [ ỹ (n ) 1 p, ỹ (n ) 2 p,..., ỹ (n ) M p ]. 2 A reformulation: u (n ) p = arg max u (n ) T p S (n ) T p u (n ) p s.t. u (n ) T p u (n ) p = 1 and u (n ) T p Ỹ (n ) p g q = 0, q = 1,..., p 1. Haiping Lu, K. N. Plataniotis, A. N. Venetsanopoulos (U of T) Uncorrelated Multilinear PCA ICML / 27

23 The UMPCA Solution. For p = 1 u (n ) 1 : the unit eigenvector of S (n ) T 1 associated with the largest eigenvalue. For p > 1 1 Let Ỹ(n ) p = [ ỹ (n ) 1 p, ỹ (n ) 2 p,..., ỹ (n ) M p ]. 2 A reformulation: u (n ) p = arg max u (n ) T p S (n ) T p u (n ) p s.t. u (n ) T p u (n ) p = 1 and u (n ) T p Ỹ (n ) p g q = 0, q = 1,..., p 1. Haiping Lu, K. N. Plataniotis, A. N. Venetsanopoulos (U of T) Uncorrelated Multilinear PCA ICML / 27

24 The UMPCA Solution. For p = 1 u (n ) 1 : the unit eigenvector of S (n ) T 1 associated with the largest eigenvalue. For p > 1 1 Let Ỹ(n ) p = [ ỹ (n ) 1 p, ỹ (n ) 2 p,..., ỹ (n ) M p ]. 2 A reformulation: u (n ) p = arg max u (n ) T p S (n ) T p u (n ) p s.t. u (n ) T p u (n ) p = 1 and u (n ) T p Ỹ (n ) p g q = 0, q = 1,..., p 1. Haiping Lu, K. N. Plataniotis, A. N. Venetsanopoulos (U of T) Uncorrelated Multilinear PCA ICML / 27

25 The UMPCA Solution. For p = 1 u (n ) 1 : the unit eigenvector of S (n ) T 1 associated with the largest eigenvalue. For p > 1 1 Let Ỹ(n ) p = [ ỹ (n ) 1 p, ỹ (n ) 2 p,..., ỹ (n ) M p ]. 2 A reformulation: u (n ) p = arg max u (n ) T p S (n ) T p u (n ) p s.t. u (n ) T p u (n ) p = 1 and u (n ) T p Ỹ (n ) p g q = 0, q = 1,..., p 1. Haiping Lu, K. N. Plataniotis, A. N. Venetsanopoulos (U of T) Uncorrelated Multilinear PCA ICML / 27

26 Solution for p > 1. Theorem The solution to the UMPCA problem (for p > 1) is the (unit-length) eigenvector corresponding to the largest eigenvalue of the following eigenvalue problem: Ψ (n ) Ψ (n ) p S (n ) T p u = λu, ) p = I In Ỹ(n p G p 1 Φ 1 p G T ) T p 1Ỹ(n p, Φ p = G T ) T p 1Ỹ(n p Ỹ (n ) p G p 1, G p 1 = [ g 1 g 2...g p 1 ] R M (p 1). I In : an identity matrix of size I n I n. Haiping Lu, K. N. Plataniotis, A. N. Venetsanopoulos (U of T) Uncorrelated Multilinear PCA ICML / 27

27 Solution for p > 1. Theorem The solution to the UMPCA problem (for p > 1) is the (unit-length) eigenvector corresponding to the largest eigenvalue of the following eigenvalue problem: Ψ (n ) Ψ (n ) p S (n ) T p u = λu, ) p = I In Ỹ(n p G p 1 Φ 1 p G T ) T p 1Ỹ(n p, Φ p = G T ) T p 1Ỹ(n p Ỹ (n ) p G p 1, G p 1 = [ g 1 g 2...g p 1 ] R M (p 1). I In : an identity matrix of size I n I n. Haiping Lu, K. N. Plataniotis, A. N. Venetsanopoulos (U of T) Uncorrelated Multilinear PCA ICML / 27

28 Outline 1 Motivation 2 The Proposed UMPCA Algorithm Tensor-to-Vector Projection Uncorrelated Multilinear PCA (UMPCA) 3 Experimental Evaluations Experimental Setup Experimental Results 4 Conclusions Haiping Lu, K. N. Plataniotis, A. N. Venetsanopoulos (U of T) Uncorrelated Multilinear PCA ICML / 27

29 Experimental Setup. Data: the FERET (face) database (subset) Maximum pose variation: 15 degrees. Minimum number of face images per subject: face images from 70 subjects. Preprocessing: manually cropped and aligned, normalized to pixels, 256 gray levels. Classification Nearest neighbor classifier. Euclidean distance measure. Performance: rank 1 identification rate. Haiping Lu, K. N. Plataniotis, A. N. Venetsanopoulos (U of T) Uncorrelated Multilinear PCA ICML / 27

30 Experimental Setup. Data: the FERET (face) database (subset) Maximum pose variation: 15 degrees. Minimum number of face images per subject: face images from 70 subjects. Preprocessing: manually cropped and aligned, normalized to pixels, 256 gray levels. Classification Nearest neighbor classifier. Euclidean distance measure. Performance: rank 1 identification rate. Haiping Lu, K. N. Plataniotis, A. N. Venetsanopoulos (U of T) Uncorrelated Multilinear PCA ICML / 27

31 Experimental Setup. Data: the FERET (face) database (subset) Maximum pose variation: 15 degrees. Minimum number of face images per subject: face images from 70 subjects. Preprocessing: manually cropped and aligned, normalized to pixels, 256 gray levels. Classification Nearest neighbor classifier. Euclidean distance measure. Performance: rank 1 identification rate. Haiping Lu, K. N. Plataniotis, A. N. Venetsanopoulos (U of T) Uncorrelated Multilinear PCA ICML / 27

32 Outline 1 Motivation 2 The Proposed UMPCA Algorithm Tensor-to-Vector Projection Uncorrelated Multilinear PCA (UMPCA) 3 Experimental Evaluations Experimental Setup Experimental Results 4 Conclusions Haiping Lu, K. N. Plataniotis, A. N. Venetsanopoulos (U of T) Uncorrelated Multilinear PCA ICML / 27

33 Recognition Results for L = 1. L: number of training samples per subject. Extreme small sample size scenario. UMPCA outperforms other three methods. Haiping Lu, K. N. Plataniotis, A. N. Venetsanopoulos (U of T) Uncorrelated Multilinear PCA ICML / 27

34 Recognition Results for L = 7. UMPCA outperforms other three methods. Haiping Lu, K. N. Plataniotis, A. N. Venetsanopoulos (U of T) Uncorrelated Multilinear PCA ICML / 27

35 Examination of Variation Captured. L = 1 L = 7 Variation captured by UMPCA is much lower (due to zero-correlation & TVP). Too low variation limits contribution in recognition. Haiping Lu, K. N. Plataniotis, A. N. Venetsanopoulos (U of T) Uncorrelated Multilinear PCA ICML / 27

36 Examination of Correlations Among Features. L = 1 L = 7 PCA and UMPCA: uncorrelated features. MPCA and TROD: correlated features. Haiping Lu, K. N. Plataniotis, A. N. Venetsanopoulos (U of T) Uncorrelated Multilinear PCA ICML / 27

37 Summary UMPCA: uncorrelated feature extraction directly from tensor objects through TVP. The solution to UMPCA: successive variance maximization & alternating projection method. Evaluation: UMPCA outperforms PCA, MPCA, TROD in unsupervised face recognition task, especially in lower dimension. Haiping Lu, K. N. Plataniotis, A. N. Venetsanopoulos (U of T) Uncorrelated Multilinear PCA ICML / 27

38 Future works Application to other unsupervised learning tasks, e.g. clustering. Investigation on design issues: initialization, projection order and termination. Combination of UMPCA features with PCA, MPCA or TROD features. Haiping Lu, K. N. Plataniotis, A. N. Venetsanopoulos (U of T) Uncorrelated Multilinear PCA ICML / 27

39 Backup Slides Basic Operations. n-mode product: (A n U)(i 1,..., i n 1, j n, i n+1,..., i N ) = i n A(i 1,..., i N ) U(j n, i n ). Scalar product: < A, B >= i 1... i N A(i 1,..., i N ) B(i 1,..., i N ). Haiping Lu, K. N. Plataniotis, A. N. Venetsanopoulos (U of T) Uncorrelated Multilinear PCA ICML / 27

40 Backup Slides Design Issues in UMPCA. Initialization: vector 1 with normalization. Projection order: from 1 to N. Termination: maximum iteration number K. Haiping Lu, K. N. Plataniotis, A. N. Venetsanopoulos (U of T) Uncorrelated Multilinear PCA ICML / 27

MPCA: Multilinear Principal Component. Analysis of Tensor Objects

MPCA: Multilinear Principal Component 1 Analysis of Tensor Objects Haiping Lu, K.N. Plataniotis and A.N. Venetsanopoulos The Edward S. Rogers Sr. Department of Electrical and Computer Engineering University