Stochastic Optimization for Deep CCA via Nonlinear Orthogonal Iterations

Size: px

Start display at page:

Download "Stochastic Optimization for Deep CCA via Nonlinear Orthogonal Iterations"

Arleen Skinner
6 years ago
Views:

Stochastic Optimization for Deep CCA via Nonlinear Orthogonal Iterations Weiran Wang Toyota Technological Institute at Chicago * Joint work

1 Stochastic Optimization for Deep CCA via Nonlinear Orthogonal Iterations Weiran Wang Toyota Technological Institute at Chicago * Joint work with Raman Arora (JHU), Karen Livescu and Nati Srebro (TTIC) 53rd Allerton Conference on Communication, Control, and Computing September 30, 2015

2 Multi-view feature learning Training data consists of samples of a D-dimensional random vector that has some natural split into two sub-vectors: [ ] x, x R Dx, y R Dy, D y x +D y = D. Natural views: audio+video, audio+articulation, text in different languages... Abstract/synthetic: word+context words, different parts of a parse tree... Task: extracting useful features/subspaces in the presence of multiple views which contain complementary information. Motivations: noise suppression, soft supervision, cross-view retrieval/generation... We assume feature dimensions have zero mean for notational simplicity.

3 Canonical correlation analysis (CCA) [Hotelling 1936] Given: data set of N paired vectors {(x 1,y 1 ),...,(x N,y N )}, which are samples of random vectors x R Dx, y R Dy. Find: direction vectors (u, v) that maximize the correlation (u,v) = argmax u,v = argmax u,v corr(u x,v y) u Σ xy v (u Σ xx u)(v Σ yy v) where Σ xy = N i=1 x iy i, Σ xx = N i=1 x ix i, Σ yy = N i=1 y iy i. Subsequent direction vectors maximize the same correlation, subject to being uncorrelated with previous directions.

4 Canonical correlation analysis (CCA) Extracting L-dimensional projections U R Dx L, V R Dy L ( ) max tr U Σ xy V U,V s.t. U Σ xx U = V Σ yy V = I. Closed-form solution obtained by SVD of Σ xy = Σ 1/2 xx Σ xy Σ 1/2 yy.

5 Canonical correlation analysis (CCA) Extracting L-dimensional projections U R Dx L, V R Dy L ( ) max tr U Σ xy V U,V s.t. U Σ xx U = V Σ yy V = I. Closed-form solution obtained by SVD of Σ xy = Σ 1/2 xx Σ xy Σ 1/2 yy. Alternative formulation Let X = [x 1,...,x N ], Y = [y 1,...,y N ]. Then CCA equivalently solves 1 min U X V Y 2 U,V 2 = 1 N U x i V 2 y i F 2 i=1 s.t. (U X)(X U) = (V Y)(Y V) = I.

6 Canonical correlation analysis (CCA) Extracting L-dimensional projections U R Dx L, V R Dy L ( ) max tr U Σ xy V U,V s.t. U Σ xx U = V Σ yy V = I. Closed-form solution obtained by SVD of Σ xy = Σ 1/2 xx Σ xy Σ 1/2 yy. Alternative formulation Let X = [x 1,...,x N ], Y = [y 1,...,y N ]. Then CCA equivalently solves 1 min U X V Y 2 U,V 2 = 1 N U x i V 2 y i F 2 i=1 s.t. (U X)(X U) = (V Y)(Y V) = I. But CCA can only find linear subspaces...

7 Deep CCA [Andrew, Arora, Bilmes and Livescu 2013] Transform the input of each view nonlinearly with deep neural networks (DNNs), such that the canonical correlation (measured by CCA) between the outputs is maximized. View 2 U f View 1 g V x y Final projection: f(x) = U f(x), g(y) = V g(y). A parametric nonlinear extension of CCA better scaling to large data than the kernel extension of CCA [Lai & Fyfe 2000, Akaho 2001, Melzer et al. 2001, Bach & Jordan 2002].

8 Deep CCA: objective and gradient Objective over DNN weights (W f,w g ) and CCA projections (U,V) ( ) max tr U FG V W f,w g,u,v s.t. U FF U = V GG V = I, where F = f(x) = [f(x 1 ),...,f(x N )] and G = g(y) = [g(y 1 ),...,g(y N )].

9 Deep CCA: objective and gradient Objective over DNN weights (W f,w g ) and CCA projections (U,V) ( ) max tr U FG V W f,w g,u,v s.t. U FF U = V GG V = I, where F = f(x) = [f(x 1 ),...,f(x N )] and G = g(y) = [g(y 1 ),...,g(y N )]. Gradient computation [Andrew, Arora, Bilmes and Livescu 2013] Let Σ fg = FG, Σ ff = FF, Σ gg = GG, and Σ fg = Σ 1/2 ff Σ fg Σ 1/2 gg = ŨΛṼ be its SVD. Then l σ l( Σ fg ) F = 2 ff F+ fg G, where ff = 1 2 Σ 1/2 ff ŨΛŨ Σ 1/2 ff, fg = Σ 1/2 ff ŨṼ Σ 1/2 gg. Gradients with respect to (W f,w g ) are computed via back-propagation.

10 Stochastic optimization of deep CCA ( ) max tr U FG V W f,w g,u,v s.t. U FF U = V GG V = I, Objective is not expectation of loss over samples due to the constraints. More difficult than PCA/PLS [Arora, Cotter, Livescu and Srebro 2012]. Exact gradient computation requires feeding-forward all data through the DNNs. Back-propagation requires large memory for large DNNs, can not be run on GPUs.

11 Stochastic optimization of deep CCA ( ) max tr U FG V W f,w g,u,v s.t. U FF U = V GG V = I, Objective is not expectation of loss over samples due to the constraints. More difficult than PCA/PLS [Arora, Cotter, Livescu and Srebro 2012]. Exact gradient computation requires feeding-forward all data through the DNNs. Back-propagation requires large memory for large DNNs, can not be run on GPUs. Question: can we do stochastic optimization for deep CCA?

12 Approach I : Large minibatches (STOL) (n) ˆΣ fg = 1/2 ˆΣ ff ˆΣ fgˆσ 1/2 gg Use a minibatch of n samples to estimate and the gradient. Works well for n large enough [Wang, Arora, Livescu and Bilmes 2015]. Total Canon. Corr STOL 100 STOL 200 STOL 300 STOL 400 STOL 500 STOL 750 STOL 1000 L-BFGS hours Does not work for small n because E [ˆΣ(n) fg ] Σ fg, due to the nonlinearities in computing Σ fg (matrix inversion, multiplication). Therefore, the gradient estimated on a minibatch is not unbiased estimate of the true gradient.

13 Alternating least squares for CCA Alternating least squares [Golub & Zha 1995]: run orthogonal iterations to obtain singular vectors (Ũ,Ṽ) of Σ fg. Input: Data matrices X, Y. Initial Ũ0 R dx L s.t. Ũ 0 Ũ0 = I. A 0 Ũ 0 Σ 1 2 ff X for t = 1,2,...,T do B t A t 1 Y ( YY ) 1 Y % Least squares regression Y At 1 B t ( ) B t B 1 2 t B t % Orthogonalize B t A t B t X ( XX ) 1 X % Least squares regression X Bt A t ( ) A t A 1 2 t A t % Orthogonalize A t end for Output: A T Ũ X, B T Ṽ Y are CCA projections as T. This procedure converges linearly under mild conditions. [Lu & Foster 2014] used a similar procedure for linear CCA with high dimensional sparse inputs.

14 Approach II : Nolinear orthogonal iterations (NOI) Choose a minibatch b of n samples at each step, and

15 Approach II : Nolinear orthogonal iterations (NOI) Choose a minibatch b of n samples at each step, and Adaptively estimate covariance matrix for orthogonalization Σ g g ρσ g g +(1 ρ) N g(y i ) g(y i ) n i b Time constant ρ [0,1). Update form is similar to that of momentum and widely used in subspace tracking. Saving Σ g g R L L costs little memory as L is usually small.

16 Approach II : Nolinear orthogonal iterations (NOI) Choose a minibatch b of n samples at each step, and Adaptively estimate covariance matrix for orthogonalization Σ g g ρσ g g +(1 ρ) N g(y i ) g(y i ) n i b Time constant ρ [0,1). Update form is similar to that of momentum and widely used in subspace tracking. Saving Σ g g R L L costs little memory as L is usually small. Replace exact linear regression with nonlinear least squares and take a gradient descent step on the minibatch 1 n f(x i ) Σ 1 2 g g g(y 2 i) min W f i b Ordinary DNN regression problem. No involved gradient.

17 Approach II : Nolinear orthogonal iterations (NOI) Choose a minibatch b of n samples at each step, and Adaptively estimate covariance matrix for orthogonalization Σ g g ρσ g g +(1 ρ) N g(y i ) g(y i ) n i b Time constant ρ [0,1). Update form is similar to that of momentum and widely used in subspace tracking. Saving Σ g g R L L costs little memory as L is usually small. Replace exact linear regression with nonlinear least squares and take a gradient descent step on the minibatch 1 n f(x i ) Σ 1 2 g g g(y 2 i) min W f i b Ordinary DNN regression problem. No involved gradient. Each step feeds-forward and back-propagates only n samples!

18 Approach II : Nolinear orthogonal iterations (NOI) Choose a minibatch b of n samples at each step, and Adaptively estimate covariance matrix for orthogonalization Σ g g ρσ g g +(1 ρ) N g(y i ) g(y i ) n i b Time constant ρ [0,1). Update form is similar to that of momentum and widely used in subspace tracking. Saving Σ g g R L L costs little memory as L is usually small. Replace exact linear regression with nonlinear least squares and take a gradient descent step on the minibatch 1 n f(x i ) Σ 1 2 g g g(y 2 i) min W f i b Ordinary DNN regression problem. No involved gradient. Each step feeds-forward and back-propagates only n samples! [Ma, Lu and Foster 2015] proposed a similar algorithm AppGrad for CCA with ρ 0.

19 Experiments: Datasets Compare three optimizers on two real-world datasets Limited-memory BFGS run with full-batch gradient Stochastic optimization with large minibatches (STOL) Nonlinear Orthogonal Iterations (NOI) Hyper-parameters (time constant ρ, minibatch size n, learning rate η) are tuned by grid search. STOL/NOI run for a maximum number of 50 epochs. Statistics of two real-world datasets. dataset training/tuning/test L DNN architectures JW11 30K/11K/9K MNIST 50K/10K/10K

20 Experiments: Effect of minibatch size n Canon. Corr JW11 MNIST 60 L-BFGS n = N 50 STOL n = STOL n = NOI n = 10 NOI n = NOI n = 50 NOI n = epoch epoch Learning curves on tuning sets with different minibatch size n. NOI works well with various small minibatch sizes. NOI gives steep improvement in the first few passes over the data.

21 Experiments: Effect of time constant ρ Canon. Corr JW11 44 n = 10 n = n = 50 n = ρ MNIST Total correlation achieved by NOI on tuning sets with different ρ. ρ NOI works well for a wide range of ρ. Beneficial to use large ρ to incorporate the previous estimate of covariance for small n.

22 Experiments: Pure stochastic optimization 30 Canon. Corr Random Init. SVD STOL n = 500 NOI n = Total correlation achieved on MNIST training sets at different ρ for linear CCA. ρ AppGrad [Ma, Lu and Foster 2015] used n O(L) with ρ = 0. Pure stochastic optimization n = 1 works well with large ρ.

23 Conclusions We have developed NOI for training deep CCA with small minibatches, alleviating the memory cost. NOI performs competitively to previous batch and stochastic optimizers.

24 Conclusions We have developed NOI for training deep CCA with small minibatches, alleviating the memory cost. NOI performs competitively to previous batch and stochastic optimizers. Future directions: Gradients of nonlinear least squares problems in NOI are not unbiased estimate of gradient of deep CCA objective. Need to better understand the convergence properties of NOI.

25 Conclusions We have developed NOI for training deep CCA with small minibatches, alleviating the memory cost. NOI performs competitively to previous batch and stochastic optimizers. Future directions: Gradients of nonlinear least squares problems in NOI are not unbiased estimate of gradient of deep CCA objective. Need to better understand the convergence properties of NOI. Thank you!

26 Bibliography I Shotaro Akaho. A kernel method for canonical correlation analysis. In Proceedings of the International Meeting of the Psychometric Society (IMPS2001), Osaka, Japan, Springer-Verlag. Galen Andrew, Raman Arora, Jeff Bilmes, and Karen Livescu. Deep canonical correlation analysis. In Sanjoy Dasgupta and David McAllester, editors, Proc. of the 30th Int. Conf. Machine Learning (ICML 2013), pages , Atlanta, GA, June Raman Arora, Andy Cotter, Karen Livescu, and Nati Srebro. Stochastic optimization for PCA and PLS. In 50th Annual Allerton Conference on Communication, Control, and Computing, pages , Montcello, IL, October Francis R. Bach and Michael I. Jordan. Kernel independent component analysis. Journal of Machine Learning Research, 3:1 48, July Gene H. Golub and Hongyuan Zha. Linear Algebra for Signal Processing, volume 69 of The IMA Volumes in Mathematics and its Applications, chapter The Canonical Correlations of Matrix Pairs and their Numerical Computation, pages Springer-Verlag, Harold Hotelling. Relations between two sets of variates. Biometrika, 28(3/4): , December P. L. Lai and C. Fyfe. Kernel and nonlinear canonical correlation analysis. Int. J. Neural Syst., 10(5): , October 2000.

27 Bibliography II Zhuang Ma, Yichao Lu, and Dean Foster. Finding linear structure in large datasets with scalable canonical correlation analysis. In Francis Bach and David Blei, editors, Proc. of the 32st Int. Conf. Machine Learning (ICML 2015), pages , Lille, France, July Thomas Melzer, Michael Reiter, and Horst Bischof. Nonlinear feature extraction using generalized canonical correlation analysis. In G. Dorffner, H. Bischof, and K. Hornik, editors, Proc. of the 11th Int. Conf. Artificial Neural Networks (ICANN 01), pages , Vienna, Austria, August Weiran Wang, Raman Arora, Karen Livescu, and Jeff Bilmes. Unsupervised learning of acoustic features via deep canonical correlation analysis. In Proc. of the IEEE Int. Conf. Acoustics, Speech and Sig. Proc. (ICASSP 15), Brisbane, Australia, April

Statistical Convergence of Kernel CCA

Statistical Convergence of Kernel CCA Kenji Fukumizu Institute of Statistical Mathematics Tokyo 106-8569 Japan fukumizu@ism.ac.jp Francis R. Bach Centre de Morphologie Mathematique Ecole des Mines de Paris,