Matrix-Tensor and Deep Learning in High Dimensional Data Analysis

Size: px

Start display at page:

Download "Matrix-Tensor and Deep Learning in High Dimensional Data Analysis"

Arabella Fields
6 years ago
Views:

1 Matrix-Tensor and Deep Learning in High Dimensional Data Analysis Tien D. Bui Department of Computer Science and Software Engineering Concordia University 14 th ICIAR Montréal July 5-7, 2017

2 Introduction Face recognition using sparse representation Background subtraction via Surveillance Matrix decomposition Videos Face Images Large-scale & High dimensional Data Text with highlighted keywords Latent Sematic Indexing 2 Predict Movie users rating rating via Matrix Factorization

3 The Matrix Problems 3

4 The Matrix Problems Matrix factorization Decomposing a matrix into product of two matrices such as PCA, SVD, Compressive Sensing and sparse dictionary learning n k n m X m A Matrix decomposition Decomposing a matrix into sum of two matrices with specific properties (e.g. sparsity, rank, etc.) Robust PCA [Candès et al. 2009] k B M (original) L (low-rank) S (sparse) 4

5 Matrix Problem Definition M = L+DS+Y; 0 is an over-complete dictionary (m<l). representing error or noise 1) Robust PCA (Candès et al.) Absence of D 2) Compressed Sensing Absence of L 3) PCA Absence of D, S (i.e. Recovery of L from M) 4) Subspace clustering problem These problems require an optimization formulation and an 5 associated norm.

Non-Convex and L p -norm Optimization An example in Compressed Sensing (CS): Aiming to find signal s sparse coefficient vector Recover exactlywith very

6 Non-Convex and L p -norm Optimization An example in Compressed Sensing (CS): Aiming to find signal s sparse coefficient vector Recover exactlywith very fewmeasurement but NP-hard Non-convex approximation Convex relaxation Solvable in poly time using LP where 0 < p< 1 Fewermeasurement than l 1 -norm but Harder to solve. 6

7 Matrix Decomposition The LpSolution of Robust PCA M = L+S 7

8 Matrix Decomposition: Robust PCA min L,S σ ( L) p + λ S p s.t. L + S = M 0 < p <1 8

9 Matrix Decomposition: Robust PCA Off-line L p RPCA min L,S s.t. d g σ j + λ g s ij j=1 ( ) L + S = M m n ij=1 ( ) where 1 3 4, 5 6 -the jthsingular value of L, 76 - an element of S Using ADMM to solve (2) Step 1 Fix L, solve for the matrix S Step 2 Fix S, solve for the matrix L Step 3 Update LagrangianmulY Alternating Direction Method of Multipliers 9 min l t,s t s.t. On-line L p RPCA n t=1 ( ( )) + λg( s t ) ( g σ L ) t (2) (3) L + S = M Perform all three steps oncefor each sample/column Compute columns of sparse matrix S by soft-thresholding on each sample Compute columns of matrix Lby updating weighted SVT incrementally Update parameter Yfor each sample Incremental Singular Value Decomposition Complexity is O(mnk), k m, i.e. linear in sample dimension and number of samples. Our paper in CVIU March 2017

10 L p -RPCA: Analysis Convergence analysis two main theorems Theorem 1: Objective function decreases monotonically Theorem 2: Converges to a stationary point [CVIU March 2017] p decreasing Objective function value and relative error (RE) of LP-RPCA algorithm on the synthetic data while varying p. (a) Shows the convergence curves of LP-RPCA algorithm. (b) Shows the performance (RE) of LP-RPCA algorithm 10 07/07/2017

11 Off-line L p -RPCA CVIU March

12 Online L p -RPCA CVIU March

13 Matrix Factorization The L p Solution of Robust SVD 13

14 Matrix Factorization: L p -SVD s.t. UU T = VV T = I Singular Values: diagonal elements of (descending order) Singular Vectors: columns of Uand rows of V T Rank of X: ,( > rank? ), equals the number of nonzero singular values Compact/truncated R F G,CA R H G, BA R G G 14

15 Robust Lp-SVD Non-convex lp approach Rewrite Objective function (4) s.t. UU T = VV T = I (4) (5) Then, (6) Linearize (6)and form the augmented Lagrangian function Using ADMM to solve (6) iteratively Robust lp-norm Singular Value Decomposition NIPSW

16 Robust Lp-SVD Rewritenobjective function: Linearize and forming Lagrangian function: Using ADMM with iterative steps: Step1 Find L Step2 Find R Step3 Find E Step4 Update Lagrangian multiplier Y Robust lp-norm Singular Value Decomposition NIPS

17 Applications of Matrix Problems Face Modelling Online Video Processing Video Inpainting Voice Music Separation 3D Structure from Motion Comparison with State of the Arts 17

18 Face Modeling : Recognition under Occlusions SR and LRA for Robust Face Recognition: two modules. 3 I J,, 4 ] ( ICPR,2014). 18

19 Face Modeling: Removing Noise, Shadows, Darkness Face modeling Face Images Clear faces Illumination CVIU March /07/2017

20 Online Video Processing Online background subtraction (Video demo) CVPR Change Detection W 14 Videos 360 x 288, 1200 frames s/frame 33fps without GPU Videos 160 x 120, 3200 frames CVIU March s/frame 200 fps without GPU 20

21 Video background subtraction Online background subtraction - Performance Time per frame decreasing 21

22 Online Video Inpainting Video inpainting (video demo) CVIU March

23 Voice Music Separation Separate a music-audio (song) stream Xinto the low-rank music part Land sparse singing voice part S Audio samples from MIR-1K Dataset Music-audio (X): Low-rank music (L): Sparse singing voice (S): Original music (L 0 ): Original singing voice (S 0 ): Huang et al., "Singing-Voice Separation From Monaural Recordings Using Robust Principal Component Analysis," ICASSP Matlabcode 23

24 3D Reconstruction from 2D Data Structure from motion X 2D points U 3D points V T Transformation 24 Bui et al. NIPS 2015

25 Comparing State-of-the-art Methods for Low-Rank and Sparse Intel Core CPU,16.00GB RAM [3] Candes et al. JACM 2011 [7] Chartrand IEEE TSP 2012 [8] Netrapalli et al. NIPS 2014 [9] Lu et al. AAAI 2015 [10] Lu et al. IEEE TIP 2015 [11] Hageet al. Comp Stat 2014 [12] Menget al. ICCV 2013 [13] Qiuet al. IEEE TIT 2014 [14] Feng et al. NIPS 2013 CVIU March

26 Tensor Decomposition 26

27 Tensor Decomposition Tucker Decomposition: approximated by a core tensor Z and three factor matrices Tensor Train Decomposition: represents a tensor compactly in terms of factors is a generalization of SVD from matrices to tensors 27

28 Tensor Decomposition (MPCA) Data is presented in tensors instead of vectors or matrices Higher-order Singular Value Decomposition (HOSVD) is used to decompose these tensors f 1 2 s 3 f d n n p s p X=Z U V V R Factor 1 Tensor X V f T n n R p p p Pixels Factor 2 Decompose U 1 Z 3 2 f V s T n s n s R Factor 3 Subjects Poses TPAMI April

29 Multilinear Principal Component Analysis (MPCA) Multilinear PCA via Kronecker product: Kronecker product f f T ( ) Z( f V T f ) s p s X = UZ V V = U V p T Reminder SVD: X = USV T low-dimensional parameters low-dimensional parameters X U Z x1,1 x1,2 1,3 x x2,1 x2,2 x2,3 = f V T s f v1 s f v2 s f 1 V f T p f 2 f 3 v p v p v p 2 subjects 3 poses TPAMI April

30 Problems with HOSVD SVD-L1 is robust against outliers and noise Italso can dealwithmissingdata Outliers SVD-L2 SVD-L1 Factor 1 Factor 3 SVD-L1 is robust against outliers Factor 2 SVD-L1 can deal with missing data TPAMI April

31 Algorithm for [ U, V] = l1svd ( X) TPAMI April

32 High Order L1-SVD The corresponding Augmented Lagrangian function): The Alternative Direction Method of Multipliers (ADMM): TPAMI April 2017 S. Boyd et al., Distributed Optimization and Statistical Learning via the Alternating Direction Method of Multipliers, Foundations and Trends in Machine Learning, Stanford, Vol. 3, No 1,

33 High Order L1-SVD Degraded Image Reconstructed Image Outliers Original Image 30% Missing Pixels PSNR = 36.3 db 50% Missing Pixels PSNR = 33.1 db TPAMI April % Missing Pixels PSNR = 27.1 db 33

34 High Order L1-SVD Original Video Foreground SVD-L1 Background TPAMI April

35 Our Works on Matrix-Tensor LRA and SR: Sparse Support Vector Machine for Pattern Recognition, Concur and Computation: Practice and Experience Non-convex Online Robust PCA: Enhance Sparsity via Lp-norm Minimization, IJCV 2017 Robust Lp-norm Singular Value Decomposition, NIPSW 2015 Are Sparse Representation and Dictionary Learning Good for Handwritten Character Recognition?, ICFHR 2014 Sparse Representation and Low-Rank Approximation for Robust Face Recognition, ICPR 2014 Tensor Decomposition Compressed Sub-manifold Multifactor Analysis, TPAMI

36 Deep Learning in Face Analysis 36

37 Our Recent Works in DL DBMs and DRBMs: 1) Deep Appearance Models: A Deep Boltzmann Machine Approach to Face Modeling, IJCV ) Beyond Principal Components: Deep Boltzmann Machines for Face Modeling, CVPR Robust DBMs: Robust Deep Appearance Models, ICPR TDRBMs: 1) Longitudinal Face Modeling via Temporal Deep Restricted Boltzmann Machines, CVPR ) Temporal Restricted Boltzmann Machines: A Survey, Pre-print RNN and CNN: Depth-Based 3D Hand Pose Tracking, ICPR

38 Deep Learning in Automatic Face Analytics System System and Framework for CVPR 2015, CVPR

39 39

40 FDDB: Face Detection Data Set and Benchmark True positive rates False positive rates 40

41 Annotated Faces in the Wild (AFW) University of California, Irvine Precisions Recalls 41

42 Precisions Recalls 42

43 43

Deep Appearance Models Active Appearance Models (AAM) cannot cope with nonlinear processes (poses, illuminations, expressions, occlusions) but DAMs can.

44 Deep Appearance Models Active Appearance Models (AAM) cannot cope with nonlinear processes (poses, illuminations, expressions, occlusions) but DAMs can. Poses Face Modeling Illumination Blur Occlusions Lowresolution Expressions CVPR 2015, CVPR 2016, ICPR 2016, and IJCV

45 = j j j j ij i j i i i i i i h a h w v b v E σ σ ) ( ) ;, ( w h v = j j j j ij i j i i i i i i h a h w v b v E σ σ ) ( ) ;, ( w h v Consider as v and as h (1) h (2) h Consider as v and as h (1) h (2) h Deep Appearance Models

46 = j j j j ij i j i i i i i i h a h w v b v E σ σ 2 2 2 ) ( ) ;, ( w h v = j j j j ij i j i i i i i i h a

46 46 = j j j j ij i j i i i i i i h a h w v b v E σ σ ) ( ) ;, ( w h v = j j j j ij i j i i i i i i h a h w v b v E σ σ ) ( ) ;, ( w h v Consider as v and as h (1) h (2) h Consider as v and as h (1) h (2) h

47 CVPR 2015, CVPR 2016 and IJCV ) Original images 2) Warped images to shape-free 3) Down-sampled by 8 (from 120x120) 4) Bicubic interpolation 5) AAM 6) DAM

48 CVPR 2015, CVPR 2016 and IJCV ) Original images 2) Warped images to shape-free 3) AAM 4) DAM

49 1) Original images 2) Warped images to shape-free 3) AAM 4) DAM CVPR 2015, CVPR 2016 and IJCV

50 CVPR 2015, CVPR 2016 and IJCV

51 CVPR 2015, CVPR 2016 and IJCV 2017 Combined effects of occlusions, poses, and illuminations 51

52 Longitudinal Face Modeling via TRBM C. N. Duong, K. Luu, K. G. Quach, and T. D. Bui. "Longitudinal Face Modeling via Temporal Deep Restricted Boltzmann Machines." In Computer Vision and Pattern Recognition (CVPR), 2016 IEEE Conference on. IEEE,

53 : Age Progression 53 CVPR 2015, CVPR 2016 and IJCV 2017

54 Depth-based 3D Hand Pose Tracking ICPR

55 Matrix and Tensor in Deep Networks 55

56 What are the Issues? Why do we care about memory? State-of-the-art deep networks are resource hungry, e.g. cannot fit in mobile devices; Up to 95% percent of parameters are in the fully connected layers; Shallow networks with huge fully connected layers can achieve almost the same accuracy as ensemble of deep CNNs 56

57 Low-rank Matrix in Deep Networks Fully-connected layers: use low-rank decomposition of the weight matrices Continuous Speech Recognition (CSR): trained with millions of parameters, with a large number of output targets. These parameters are in the final fully connected layers Acoustic and language modeling: Low-rank matrix has shown astonishing results. The number of parameters is reduced by up to 50% with the same reduction in training time, without loss in recognition accuracy, compared to a full-rank representation Low-rank Matrix Factorization for Deep Neural Network Training with High Dimensional Output, T.N. Sainath et al

58 Tensor Train in Deep Neural Nets Tensor Train (or linear tensor network) to represent the dense weight matrices of the fully-connected layers such that (1) the number of parameters is reduced by a huge factor; and (2) the expressive power of the layer is preserved. For Very Deep networks the compression factor of the dense weight matrix of a fully-connected layer can be up to times leading to the compression factor of the whole network up to 7 times. Deep multi-task representation learning where sparse representations shared across multiple tasks. Tensorizing Neural Networks, A. Novikov et al Deep Multi-task Representation Learning: A Tensor Fac. Approach, Y. Yang et al ICLR 58

59 How to find Tensor Train? Special Cases: Analytical formulas. Medium size Tensors: Exact algorithm based on SVD For large tensors (e.g. 2^50): Approximate algorithms that look at a fraction of the tensor elements There are Python codes available Neural networks use fully-connected layers. The matrix W is of millions parameters. Store and train the matrix W in the TTformat can speed up the training VGG-16 is trained on more than a million images in ImageNet and can classify images into 1000 object categories. For VGG-16 net we compressed matrix to 320 parameters without loss of accuracy 59

60 Thank you 60

Structured matrix factorizations. Example: Eigenfaces

Structured matrix factorizations Example: Eigenfaces An extremely large variety of interesting and important problems in machine learning can be formulated as: Given a matrix, find a matrix and a matrix