Online Learning for Sparse PCA in High Dimensions: Exact Dynamics and Phase Transitions

Transcription

1 Online Learning for Sarse PCA in High Dimensions: Eact Dynamics and Phase Transitions Chuang Wang and Yue M. Lu John A. Paulson School of Engineering and Alied Sciences Harvard University Abstract We study the dynamics of an online algorithm for learning a sarse leading eigenvector from samles generated from a siked covariance model. This algorithm combines the classical Oja s method for online PCA with an element-wise nonlinearity at each iteration to romote sarsity. In the high-dimensional limit, the joint emirical measure of the underlying sarse eigenvector and its estimate rovided by the algorithm is shown to converge weakly to a deterministic, measure-valued rocess. This scaling limit is characterized as the unique solution of a nonlinear PDE, and it rovides eact information regarding the asymtotic erformance of the algorithm. For eamle, erformance metrics such as the cosine similarity and the misclassification rate in sarse suort recovery can be obtained by eamining the limiting dynamics. A steady-state analysis of the nonlinear PDE also reveals an interesting hase transition henomenon. Although our analysis is asymtotic in nature, numerical simulations show that the theoretical redictions are accurate for moderate signal dimensions. I. INTRODUCTION Consider the siked covariance model [], where we are given a sequence of -dimensional samle vectors y, y,... that are distributed according to ω y k = c kξ + a k. Here, ξ is an unknown vector in R, c k N,, a k N, I, and ω is a ositive quantity secifying the signalto-noise ratio SNR; c i, a i and c j, a j are indeendent for i j. In this aer, we analyze the eact dynamics of an online incremental algorithm for estimating ξ in the highdimensional limit. The model in arises in the theoretical study of rincial comonent analysis PCA, an imortant statistical tool in eloratory data analysis, visualization and dimension reduction. A standard method to estimate ξ is to comute the leading eigenvector of the samle covariance matri Σ = n n k= y ky T k. For fied and when the number of samles n tends to infinity, the eigenvector is a consistent estimator of ξ u to a normalization constant. However, in the regime where and n are both large and comarable in size, the estimate given by the eigenvector is no longer consistent [], [3]. To address this issue, a flurry of work under the name of sarse PCA has eloited the sarsity structure of ξ see, e.g., This work was suorted in art by the NSF under grant CCF-394 and by ARO under grant W9NF [3] [5]. In addition to otentially imroving the estimate of ξ, sarse PCA generates a more arsimonious and interretable reresentation, using a small subset of feature variables to elain the original data. The natural formulation of sarse PCA leads to nonconve otimization roblems [3] [5]. Conve relaations via semiinite rogramming SDP [6], [7] are ossible, but the comutational and storage cost of SDP may become rohibitive when the dimensionality is high. Many efficient algorithms have been roosed to solve sarse PCA, in both offline [3], [8] [] and online [] [5] settings. In the latter case, which is the setting we study in this aer, samle vectors {y k } arrive sequentially in an infinite stream; as soon as a new samle vector or a small batch of them has arrived, an online algorithm comutes an instantaneous udate to its estimate of ξ. Since they only kee and oerate on small sets of current samles, online algorithms are memory and comutationally efficient. Moreover, as they rovide estimates on-the-fly, online algorithms are well-suited to dynamic scenarios where the rincial comonent vectors can be time-varying. In this aer, we analyze an online sarse PCA algorithm that combines the classical Oja s method [6] with an element-wise nonlinearity e.g., soft-thresholding at each iteration to romote sarsity see Section II for the eact form. Secifically, let k be the estimate of ξ given by the algorithm uon receiving the kth samle; let i k and ξi denote the ith comonent of each vector. Also, ine the joint emirical measure of k and ξ as µ k, ξ = δ i k, ξ ξ i. i= Note that µ k, ξ is a random element in MR, the sace of robability measures on R. As the main result of this work, we show that, as and with suitable time-rescaling, the sequence of emirical measures { µ k, ξ} converges weakly to a deterministic measure-valued rocess µ t, ξ. Moreover, this limiting measure µ t, ξ is the unique solution of a nonlinear artial differential equation PDE. The deterministic scaling limit as secified by the PDE and its solution rovides a wealth of information regarding the erformance of the online sarse PCA algorithm. For eamle, the limiting value of the cosine similarity Q k = T k ξ k ξ 3

2 at any ste k can be easily obtained by comuting the eectation Eξ with resect to the limiting measure µ t, ξ. More involved questions, such as the misclassification rate in sarse suort recovery, can also be answered by eamining µ t, ξ. Finally, studying the PDE in its steady-state leads to an eact characterization of the long-time behavior of the online sarse PCA algorithms. This steady-state analysis also uncovers a hase transition henomenon: the erformance of the algorithm can ehibit markedly different behaviors deending on the arameter settings and SNR values. The rest of the aer is organized as follows. In Section II, we give the details of the online sarse PCA algorithm that we analyze in this work. The scaling limit of the algorithm is resented in Section III. As a secial case, we study in Section III-B the classical Oja s method and derive an analytical eression characterizing the limiting cosine similarity between its estimates and ξ. Finally, a steady-state analysis and an associated hase transition henomenon are discussed in Section IV. II. ONLINE ALGORITHM FOR SPARSE PCA We consider the online setting, where samle vectors {y k } arrive sequentially. We assume that the samles are generated by the siked covariance model in with a single leading eigenvector ξ. We further assume that each element of ξ is an i.i.d. samle drawn from a miture distribution πξ = ρδξ + ρ uξ, 4 where ρ, ] is a arameter controlling the sarsity level, and uξ is a density function such that ξ uξ dξ = /ρ. The receding requirement makes sure that ξ / as the dimension. An eamle of 4 is the standard Bernoulli-Gaussian distribution. By choosing uξ = [δξ / ρ + δξ + / ρ]/, the distribution in 4 can also describe the sarse signal model considered in [7]. In this work, we analyze a simle recursive algorithm for estimating ξ from the stream of samles {y k }. The algorithm starts from some initial estimate. Uon receiving the kth data samle y k, it udates its estimate as follows: k = k + τ/ y k y T k k k = η k / η k. Here, τ > is the ste size, and η is an element-wise nonlinear maing taking the form 5 η = φ, 6 for some iecewise smooth function φ : R R. Clearly, the method is online incremental: it rocesses one samle at a time. Once a samle has been rocessed, it will be discarded and never used again. The udate stes in 5 as well as the eression in 6 need some elanations. First, we note that, without the nonlinear maing i.e., by setting η =, the recursions in 5 are eactly the original Oja s method [6] for online PCA. The nonlinearity 6 in η is introduced to romote sarsity of the estimates. To see this, we consider an otimization formulation for sarse PCA in the offline setting: = arg min = T Σ + Φ i, 7 where Σ is the oulation or samle covariance matri, and Φ is an element-wise enalty function that favors sarse solutions. For eamle, Φ = λ for lasso-tye enalizations; or we can choose Φ = λ + λ for the elastic net [5]. To solve 7, we use a roimal gradient method [7] followed by a rojection onto the shere of radius : i= k = k + τ/σ k k = ro τφ/ k ro τφ/ k, where ro τφ/ denotes the roimal oerator of the function τ Φ/. Relacing the covariance matri Σ by its instantaneous and noisy version y k y T k and using the aroimation ro τφ/ τ Φ/ see, e.g., [7,. 38] for a justification of this aroimation which holds for large, we reach our algorithm in 5 as well as the form given in 6. Eamle : Consider a lasso-tye enalization in 7 where Φ = β τ for some β >. The associated roimal oerator is the standard soft-thresholding function with arameter β/, which can be aroimated, for large, as ro τφ/ β sgn. This corresonds to choosing φ = β sgn in 6. In what follows, we refer to this articular variant of the algorithm as Oja s algorithm with iterative soft thresholding OIST for short. III. DYNAMICS IN HIGH DIMENSIONS: SCALING LIMITS In what follows, we analyze the dynamics of the online sarse PCA algorithm in 5 in the large limit. The central object in our analysis is the emirical measure µ k, ξ as ined in. Here, the subscrit k indicates the iteration ste, and the suerscrit makes elicit the deendence of the measure on the dimension. The measure µ k contains a great deal of information about the algorithm. For eamle, using the notation f, µ k = f i k, ξ i, i for a test function f, ξ, we can write the cosine similarity ined in 3 as Q k = ξ, µ k / ξ, µ k. Similarly, more involved quantities such as the misclassification rate in sarse suort recovery can also be written in terms of µ k.

3 A. The Main Convergence Result To establish the scaling limit of µ k, we first embed the discrete-time sequence in continuous-time by ining.8.6 OIST: simulation PDE: theory.8.6 µ t = µ t, where is the floor function. Similarly, we can ine Q t as the continuous-time rescaled version of Q k. Note that this tye of time embedding and rescaling is standard in studying the convergence of stochastic rocesses [8]. Some technicalities before we move on: since the emirical measure is random, µ t is a iecewise-constant càdlàg rocess taking values in MR, the sace of robability measures on R. In short, µ t is a random element in DR +, MR, for which the notion of weak convergence is well-ined. See, e.g., [9]. Theorem : Suose that µ, the emirical measure at time k =, converges weakly to a deterministic measure µ MR and that Q = ξ, µ. Then, as, the measure-valued stochastic rocess µ t converges weakly to a deterministic rocess µ t, characterized as the unique solution to the following nonlinear PDE given in the weak form: for any ositive, bounded and C 3 test function f, ξ, where f, µ t = f, µ + + τ t t Q t = ξ dµ t, Γ, ξ, Q s, R s f, µ s ds + ωq s f, µ s ds, R t = with φ being the function introduced in 6, and 8 φ dµ t ; 9 Γ, ξ, Q, R = τωqξ φ [τωq R+ τ ] +ωq. Remark : The deterministic measure-valued rocess µ t, ξ characterizes the eact dynamics of the online sarse PCA algorithm in 5 in the high-dimensional limit. The nonlinear PDE 8 secifies the time evolution of µ t, ξ. Note that 8 is resented in the weak form. If the strong, density valued solution eists, then it must satisfy t P t ξ = [ Γ, ξ, Qt, R t P t ξ ] + τ + ωq t P t ξ, where we use P t ξ to denote the conditional density of given ξ at time t. The joint density can then be comuted as P t, ξ = P t ξπξ, where πξ is the marginal density ined in 4. Remark : For each ξ, the PDE resembles a Fokker- Planck equation [] describing the time-evolution of the robability density associated with a article undergoing a driftdiffusion rocess in one satial dimension. There is, however, one imortant distinction: the PDEs associated with different values of ξ are couled via the quantities Q t and R t, which Fig.. Theory v.s. simulations. The figures show comarisons between the limiting conditional densities P t ξ as redicted by the PDE and the emirical densities obtained from Monte Carlo simulations. To row: t = ; bottom row: t = 5; left column: ξ = ; and right column: ξ = / ρ. See Eamle for details of the eeriment. themselves deend on the current densities P t ξ. To see this, we rewrite 9 as Q t = E ξ ξ P t ξ d R t = E ξ φp t ξ d, 3 where E ξ denotes the eectation with resect to the variable ξ drawn from the rior distribution πξ. Proosition : Under the same assumtions of Theorem, the stochastic rocess Q t = Q t converges weakly, as, to the deterministic function Q t ined in 9. Remark 3: We note that Q t describes the time-evolution of the cosine similarity 3 between the estimate given by the algorithm and the unknown vector ξ. This result shows that the dynamics of Q t converges to a deterministic curve Q t, which can be comuted from the limiting measure µ t. Eamle : The roofs of Theorem and Proosition will be resented elsewhere. Here, we verify the accuracy of the theoretical redictions made in them via numerical simulations. In our eeriment, we generate a vector ξ whose comonents are i.i.d. and drawn from a marginal distribution πξ = ρδξ + ρδξ / ρ. The sarsity level is set to ρ =.5. Starting from a random initial estimate with i.i.d. entries drawn from a normal distribution N,, we use the OIST version of the online sarse PCA algorithm see Eamle to estimate ξ. The dimension is set to =,, and the other arameters are τ =.5, β =.7, and ω =. In Figure, we comare the redicted limiting conditional densities P t ξ = and P t ξ = / ρ against the emirical densities observed in the simulations, at two different times t = and t = 5. The PDE in is solved numerically. We can see from the figure that the limiting densities given by the theory rovide accurate redictions for the simulation

4 Qt OIST: simulation PDE: theory Steady-State. Oja s: simulation ODE: theory 5 5 t Fig.. The comarison between the analytical redictions of the cosine similarity Q t and Monte Carlo simulations. For OIST, the theoretical curve is comuted by using ; for Oja s method, we use the closed-form formula in 4. The theoretical redictions are lotted as dashed and solid lines, whereas the average values of Monte Carlo simulations are lotted as squares and circles. The error bars show confidence intervals of ± standard deviations. The black dotted line indicates the theoretical rediction of the steady-state given by the solution of the fied-oint equations in 9. results. In Figure, we verify the limiting form of the cosine overla Q t as given in. For simulations, we average over indeendent instances of OIST, and lot the mean values and confidence intervals ± standard deviations. Again, we can see that the asymtotic results match with simulation data very well. Also shown in the figure are results for the standard Oja s method, for which we can obtain a closed-form analytical formula for Q t. This is the focus of the following subsection. B. The Nonsarse Case: Oja s Method As mentioned earlier, the classical Oja s method [6] can be viewed as a secial case of the algorithm in 5. It corresonds to setting φ = in 6, i.e., the algorithm does not aly the nonlinear maing η. In this case, the limiting PDE can be converted to a linear Fokker-Planck equation for the Ornstein-Uhlenbeck rocess, for which analytical solutions eist. For brevity, we omit discussions of this analytical solution of the PDE. Instead, we show a related result regarding the cosine similarity Q t, which is an imortant figure of merit for the algorithm. Proosition : For Oja s method, assume that we start the algorithm with a nonzero cosine similarity, i.e., Q Q as. Then the dynamics of the cosine similarity Q t Q t, where Q t is given by α Q [α + t = α Q α t + Q ] α e αt if α if α =. 4 Here, α = τω + τ and α = τω τ. Proof sketch: We substitute f, ξ = ξ into the weak form 8 of the limiting PDE. The left-hand side is then eactly Q t. Using the facts that E ξ ξ =, φt =, and after some maniulations, we can simlify the right-hand side of 8 and get Q t = Q + t α Q 3 s + α Q s ds. Solving this ordinary differential equation leads to 4. In the long-time limit, we have lim t Q t = ma {, ω τ ω + τ }. 5 This result indicates that for any finite ste size τ >, Oja s method for online PCA cannot reach erfect estimation i.e., Q = even with infinite number of samles. Moreover, the formula also oints out a simle hase transition henomenon: when τ > ω, the estimates obtained by the algorithm will be uncorrelated with ξ. IV. STEADY STATE ANALYSIS AND PHASE TRANSITIONS In this section, we study the long-time limit of OIST for sarse PCA. This steady-state analysis reveals an interesting hase transition henomenon associated with OIST, which we also briefly discuss. In the long-time limit, uon reaching the steady-state, the left-hand side of becomes. It follows that the steadystate density functions satisfy the equation τ + ωq P ξ = Γ, ξ, Q, RP ξ, 6 where P ξ, Q, R are the steady-state versions of P t ξ, Q t and R t, resectively. Solving 6 and eanding Γ according to its inition in, we find the steady-state conditional density in the form of a Boltzmann distribution: P ξ = Z ξ e where Z ξ is the artition function, hq, R + Φ τωqξ gq gq = τ + ωq / hq, R = τωq R + gq /, 7 8 and Φ is an antiderivative of φ. Note that Φ can be any such antiderivative, since any constant added to Φ will be absorbed into the normalization constant Z ξ. It is imortant to emhasize that 7 is only an imlicit inition of the steady-state distribution. This is because the eression relies on two constants Q and R, whose values are determined by the self-consistent equations and 3 with t involving P ξ. In what follows, we focus on OIST as discussed in Eamle. Here, φ = β sgn, and thus we can set Φ = β. It follows that the eonent in 7 is a iecewise quadratic olynomial. This convenient form allows us to further simlify the right-hand sides of and 3. After some maniulations which are omitted here, we can obtain the following fiedoint equations for determining Q and R: gq Q = hq, R E ξ gq R = β hq, R E ξ ξ z +fz + z fz fz + + fz, π z +fz + z fz fz + + fz, 9 where gq, hq, R are the functions ined in 8, z ± = gqhq, R β ± τωξq /, and f is the scaled com- limentary error function ined as f = { } π e One can check that Q =, R = τ is always a solution to the fied-oint equations 9. We call any such solution with e z dz.

5 .5 ω =.5 ω =. ω =.5 ω = Q OIST Oja s ω Fig. 3. Steady-state distributions and hase transitions. Left-hand side: The steady-state densities P ξ = / ρ at different SNR values. Right-hand side: Theoretical redictions of the steady-state cosine overla Q as a function of the SNR arameter ω. Black solid line: theoretical rediction for OIST; red dots: simulation results; black dashed line: theoretical rediction for Oja s method; blue squares: simulation results. Q = an uninformative solution, since it corresonds to a final estimate that is uncorrelated with ξ. It is also revealing to eamine the corresonding steady-state distributions. Substituting Q, R into 7, we find that, for any ξ, the conditional density is of the form P ξ = β β e τ τ. Since P ξ does not deend on ξ, the variables and ξ are indeendent; thus, the estimate rovided by the algorithm contains no information about ξ in the long-time limit. In the low SNR regime, such uninformative fied-oints are the only solutions to 9. The situation imroves when we increase the SNR arameter ω. At a certain critical value ω c, a nontrivial fied oint {Q, R } with Q emerges. This corresonds to the case when the estimate becomes informative. We will resent more detailed analysis of this hase transition henomenon elsewhere. In what follows, we illustrate it using a numerical eamle. We consider OIST at different SNR values. The other arameters in the algorithm are the same as those used in Eamle. The left-side of Figure 3 shows the limiting steady-state conditional densities P ξ = / ρ for increasing values of the SNR arameter ω. At a low SNR value ω =.5, we get the zero-mean uninformative Lalace distribution in. As ω increases, the modes of the conditional densities move towards / ρ, starting to reveal information about ξ. In the right-side of Figure 3, we show the steady-state values of the cosine overla Q as a function of ω. A clear hase transition aears at a critical value ω c. The theoretical rediction the solid line in the figure, obtained by numerically solving the fied-oint equations 9, matches very well with Monte Carlo simulations of the algorithm shown as red dots. Also shown in the figure are the results for Oja s method, with its theoretical rediction given by 5. Comaring OIST with Oja s method, we see that OIST has a lower hase transition threshold and that it also achieves a higher steady-state value for Q. This imrovement in erformance can be attributed to the fact that OIST eloits the sarsity structure of ξ via iterative thresholding. V. CONCLUSION We analyzed the dynamics of an online sarse PCA algorithm in the high-dimensional limit. The joint emirical measure of the underlying sarse eigenvector and its estimate as rovided by the algorithm converges weakly to a deterministic rocess, characterized as the unique solution of a nonlinear PDE. This scaling limit rovides eact information regarding the asymtotic erformance of the algorithm. As a secial case, we derived a closed-form eression for the limiting dynamics of the cosine similarity associated with Oja s method, a classical algorithm for online PCA. We also studied the steady-state of the nonlinear PDE and observed a hase transition henomenon. The theoretical framework in this work is general. It aves the way towards understanding the dynamics of other online algorithms for various high-dimensional estimation roblems. The theoretical analysis also rovides insights and can lead to more rinciled ways of otimizing arameters in the algorithm to further imrove erformance. REFERENCES [] I. M. Johnstone, On the distribution of the largest eigenvalue in rincial comonents analysis, Ann. Stat., vol. 9, no., , Ar.. [] B. Nadler, Finite samle aroimation results for rincial comonent analysis: A matri erturbation aroach, Ann. Stat., vol. 36, no. 6, , 8. [3] I. M. Johnstone and A. Y. Lu, On consistency and sarsity for rincial comonents analysis in high dimensions, J. Am. Stat. Assoc., vol. 4, no. 486, , Jun. 9. [4] I. T. Jolliffe, N. T. Trendafilov, and M. Uddin, A modified rincial comonent technique based on the LASSO, J. Com. Grah. Stat., vol., no. 3, , Se. 3. [5] H. Zou, T. Hastie, and R. Tibshirani, Sarse rincial comonent analysis, J. Com. Grah. Stat., vol. 5, no., , 6. [6] A. d Asremont, L. El Ghaoui, M. I. Jordan, and G. R. G. Lanckriet, A direct formulation for sarse PCA using semiinite rogramming, SIAM Rev., vol. 49, no. 3, , Jan. 7. [7] A. A. Amini and M. J. Wainwright, High-dimensional analysis of semiinite relaations for sarse rincial comonents, Ann. Stat., vol. 37, no. 5B, , Oct. 9. [8] H. Shen and J. Z. Huang, Sarse rincial comonent analysis via regularized low rank matri aroimation, J. Maultivar. Anal., vol. 99, no. 6,. 5 34, Jul. 8. [9] M. Journée, Y. Nesterov, P. Richtárik, and R. Seulchre, Generalized ower method for sarse rincial comonent analysis, J. Mach. Learn. Res., vol., no ,. [] Z. Ma, Sarse rincial comonent analysis and iterative thresholding, Ann. Stat., vol. 4, no.,. 77 8, Ar. 3. [] Y. Deshande and A. Montanari, Information-theoretically otimal sarse PCA, in IEEE International Symosium on Information Theory, 4. [] J. Mairal, F. Bach, J. Ponce, and G. Sairo, Online learning for matri factorization and sarse coding, J. Mach. Learn. Res., vol.,. 9 6,. [3] R. Arora, A. Cotter, K. Livescu, and N. Srebro, Stochastic otimization for PCA and PLS, in Proc. 5th Annual Allerton Conference on Communication, Control, and Comuting Allerton, Oct.. [4] A. Balsubramani, S. Dasguta, and Y. Freund, The fast convergence of incremental PCA, in Adv. Neural Inf. Process. Syst., 3. [5] W. Yang and H. Xu, Streaming sarse rincial comonent analysis, in Proceedings of the 3nd International Conference on Machine Learning ICML-5, 5, [6] E. Oja and J. Karhunen, On stochastic aroimation of the eigenvectors and eigenvalues of the eectation of a random matri, J. Math. Anal. Al., vol. 6, no., , 985. [7] N. Parikh and S. Boyd, Proimal Algorithms, Foundations and Trends in Otimization, vol., no. 3, Jan. 4. [8] P. Billingsley, Convergence of robability measures, nd ed. New York: Wiley, 999. [9] O. Kallenberg, Foundations of modern robability, nd ed. Sringer,. [] H. Risken, The Fokker-Planck equation: Methods of solution and alications, nd ed. New York: Sringer-Verlag, 996.