Non-convex Robust PCA: Provable Bounds
|
|
|
- Pearl Eugenia Black
- 5 years ago
- Views:
Transcription
1 Non-convex Robust PCA: Provable Bounds Anima Anandkumar U.C. Irvine Joint work with Praneeth Netrapalli, U.N. Niranjan, Prateek Jain and Sujay Sanghavi.
2 Learning with Big Data High Dimensional Regime Missing observations, gross corruptions, outliers, ill-posed problems. Needle in a haystack: finding low dimensional structures in high dimensional data. Principled approaches for finding low dimensional structures?
3 PCA: Classical Method Denoising: find hidden low rank structures in data. Efficient computation, perturbation analysis.
4 PCA: Classical Method Denoising: find hidden low rank structures in data. Efficient computation, perturbation analysis.
5 PCA: Classical Method Denoising: find hidden low rank structures in data. Efficient computation, perturbation analysis. Not robust to even a few outliers
6 Robust PCA Problem Find low rank structure after removing sparse corruptions. Decompose input matrix as low rank + sparse matrices. M L S M R n n, L is low rank and S is sparse. Applications in computer vision, topic and community modeling.
7 History Heuristics without guarantes Multivariate trimming [Gnanadeskian+ Kettering 72] Random sampling [Fischler+ Bolles81]. Alternating minimization [Ke+ Kanade03]. Influence functions [de la Torre + Black 03] Convex methods with Guarantees Chandrasekharan et. al, Candes et. al 11: seminal guarantees. Hsu et. al 11, Agarwal et. al 12: further guarantees. (Variants) Xu et. al 11: Outlier pursuit, Chen et. al 12: community detection.
8 Why is Robust PCA difficult? M L S No identifiability in general: Low rank matrices can also be sparse and vice versa. Natural constraints for identifiability? Low rank matrix is NOT sparse and viceversa. Incoherent low rank matrix and sparse matrix with sparsity constraints. Tractable methods for identifiable settings?
9 Why is Robust PCA difficult? M L S No identifiability in general: Low rank matrices can also be sparse and vice versa. Natural constraints for identifiability? Low rank matrix is NOT sparse and viceversa. Incoherent low rank matrix and sparse matrix with sparsity constraints. Tractable methods for identifiable settings?
10 Convex Relaxation Techniques (Hard) Optimization Problem, given M R n n min L,S Rank(L)+γ S 0, M = L+S. Rank(L) = {#σ i (L) : σ i (L) 0}, S 0 = {#S(i,j) : S(i,j) 0} are not tractable.
11 Convex Relaxation Techniques (Hard) Optimization Problem, given M R n n min L,S Rank(L)+γ S 0, M = L+S. Rank(L) = {#σ i (L) : σ i (L) 0}, S 0 = {#S(i,j) : S(i,j) 0} are not tractable. Convex Relaxation min L,S L +γ S 1, M = L+S. L = i σ i(l), S 1 = i,j S(i,j) are convex sets. Chandrasekharan et. al, Candes et. al 11: seminal works.
12 Other Alternatives for Robust PCA? min L,S L +γ S 1, M = L+S. Shortcomings of convex methods
13 Other Alternatives for Robust PCA? min L,S L +γ S 1, M = L+S. Shortcomings of convex methods Computational cost: O(n 3 /ǫ) to achieve error of ǫ Requires SVD of n n matrix. Analysis: requires dual witness style arguments. Conditions for success usually opaque.
14 Other Alternatives for Robust PCA? min L,S L +γ S 1, M = L+S. Shortcomings of convex methods Computational cost: O(n 3 /ǫ) to achieve error of ǫ Requires SVD of n n matrix. Analysis: requires dual witness style arguments. Conditions for success usually opaque. Non-convex alternatives?
15 Proposal for Non-convex Robust PCA min L,S S 0, s.t. M = L+S, Rank(L) = r
16 Proposal for Non-convex Robust PCA min L,S S 0, s.t. M = L+S, Rank(L) = r A non-convex heuristic (AltProj) Initialize L,S = 0 and iterate: L P r (M S) and S H ζ (M L). P r ( ): rank-r projection. H ζ ( ): thresholding with ζ. Computationally efficient: each operation is just a rank-r SVD or thresholding.
17 Proposal for Non-convex Robust PCA min L,S S 0, s.t. M = L+S, Rank(L) = r A non-convex heuristic (AltProj) Initialize L,S = 0 and iterate: L P r (M S) and S H ζ (M L). P r ( ): rank-r projection. H ζ ( ): thresholding with ζ. Computationally efficient: each operation is just a rank-r SVD or thresholding. Any hope for proving guarantees?
18 Observations regarding non-convex analysis Challenges Multiple stable points: bad local optima, solution depends on initialization. Method may have very slow convergence or may not converge at all!
19 Observations regarding non-convex analysis Challenges Multiple stable points: bad local optima, solution depends on initialization. Method may have very slow convergence or may not converge at all! Non-convex Projections vs. Convex Projections Projections on to non-convex sets: NP-hard in general. Projections on to rank and sparse sets: tractable. Less information than convex projections: zero-order conditions. P(M) M Y M, Y C(Non-convex), P(M) M 2 Y M,P(M) M, Y C(Convex).
20 Non-convex success stories Classical Result PCA: Convergence to global optima!
21 Non-convex success stories Classical Result PCA: Convergence to global optima! Recent results Tensor methods (Anandkumar et. al 12, 14): Local optima can be characterized in special cases.
22 Non-convex success stories Classical Result PCA: Convergence to global optima! Recent results Tensor methods (Anandkumar et. al 12, 14): Local optima can be characterized in special cases. Dictionary learning (Agarwal et. al 14, Arora et. al 14): Initialize using a clustering style method and do alternating minimization.
23 Non-convex success stories Classical Result PCA: Convergence to global optima! Recent results Tensor methods (Anandkumar et. al 12, 14): Local optima can be characterized in special cases. Dictionary learning (Agarwal et. al 14, Arora et. al 14): Initialize using a clustering style method and do alternating minimization. Matrix completion/phase retrieval: (Netrapalli et. al 13) Initialize with PCA and do alternating minimization.
24 Non-convex success stories Classical Result PCA: Convergence to global optima! Recent results Tensor methods (Anandkumar et. al 12, 14): Local optima can be characterized in special cases. Dictionary learning (Agarwal et. al 14, Arora et. al 14): Initialize using a clustering style method and do alternating minimization. Matrix completion/phase retrieval: (Netrapalli et. al 13) Initialize with PCA and do alternating minimization. (Somewhat) common theme Characterize basin of attraction for global optimum. Obtain a good initialization to land in the ball.
25 Non-convex Robust PCA A non-convex heuristic (AltProj) Initialize L,S = 0 and iterate: L P r (M S) and S H ζ (M L). Observations regarding Robust PCA Projection on to rank and sparse subspaces: non-convex but tractable: SVD and hard thresholding. But alternating projections: challenging to analyze
26 Non-convex Robust PCA A non-convex heuristic (AltProj) Initialize L,S = 0 and iterate: L P r (M S) and S H ζ (M L). Observations regarding Robust PCA Projection on to rank and sparse subspaces: non-convex but tractable: SVD and hard thresholding. But alternating projections: challenging to analyze Our results for (a variant of) AltProj Guaranteed recovery of low rank L and sparse part S. Match the bounds for convex methods (deterministic sparsity). Reduced computation: only require low rank SVDs!
27 Non-convex Robust PCA A non-convex heuristic (AltProj) Initialize L,S = 0 and iterate: L P r (M S) and S H ζ (M L). Observations regarding Robust PCA Projection on to rank and sparse subspaces: non-convex but tractable: SVD and hard thresholding. But alternating projections: challenging to analyze Our results for (a variant of) AltProj Guaranteed recovery of low rank L and sparse part S. Match the bounds for convex methods (deterministic sparsity). Reduced computation: only require low rank SVDs! Best of both worlds: reduced computation with guarantees!
28 Outline 1 Introduction 2 Analysis 3 Experiments 4 Robust Tensor PCA 5 Conclusion
29 Toy example: Rank-1 case M = L +S, L = u (u ) Non-convex method (AltProj) Initialize L,S = 0 and iterate: L P 1 (M S) and S H ζ (M L). P 1 ( ): rank-1 projection. H ζ ( ): thresholding.
30 Toy example: Rank-1 case M = L +S, L = u (u ) Non-convex method (AltProj) Initialize L,S = 0 and iterate: L P 1 (M S) and S H ζ (M L). P 1 ( ): rank-1 projection. H ζ ( ): thresholding. Immediate Observations First PCA: L P 1 (M).
31 Toy example: Rank-1 case M = L +S, L = u (u ) Non-convex method (AltProj) Initialize L,S = 0 and iterate: L P 1 (M S) and S H ζ (M L). P 1 ( ): rank-1 projection. H ζ ( ): thresholding. Immediate Observations First PCA: L P 1 (M). Matrix perturbation bound: M L 2 O( S )
32 Toy example: Rank-1 case M = L +S, L = u (u ) Non-convex method (AltProj) Initialize L,S = 0 and iterate: L P 1 (M S) and S H ζ (M L). P 1 ( ): rank-1 projection. H ζ ( ): thresholding. Immediate Observations First PCA: L P 1 (M). Matrix perturbation bound: M L 2 O( S ) If S 1, no progress!
33 Toy example: Rank-1 case M = L +S, L = u (u ) Non-convex method (AltProj) Initialize L,S = 0 and iterate: L P 1 (M S) and S H ζ (M L). P 1 ( ): rank-1 projection. H ζ ( ): thresholding. Immediate Observations First PCA: L P 1 (M). Matrix perturbation bound: M L 2 O( S ) If S 1, no progress! Exploit incoherence of L?
34 Rank-1 Analysis Contd. M = L +S, L = u (u ) Non-convex method (AltProj) Initialize L,S = 0 and iterate: L P 1 (M S) and S H ζ (M L).
35 Non-convex method (AltProj) Initialize L,S = 0 and iterate: Rank-1 Analysis Contd. M = L +S, L = u (u ) L P 1 (M S) and S H ζ (M L). Incoherence of L L = u (u ) and u µ n and L µ2 n.
36 Non-convex method (AltProj) Initialize L,S = 0 and iterate: Rank-1 Analysis Contd. M = L +S, L = u (u ) L P 1 (M S) and S H ζ (M L). Incoherence of L L = u (u ) and u µ n and L µ2 n. Solution for handling large S First threshold M before rank-1 projection. Ensures large entries of S are identified.
37 Non-convex method (AltProj) Initialize L,S = 0 and iterate: Rank-1 Analysis Contd. M = L +S, L = u (u ) L P 1 (M S) and S H ζ (M L). Incoherence of L L = u (u ) and u µ n and L µ2 n. Solution for handling large S First threshold M before rank-1 projection. Ensures large entries of S are identified. Choose threshold ζ 0 = 4µ2 n.
38 Non-convex method (AltProj) Rank-1 Analysis Contd. M = L +S, L = u (u ) Initialize L = 0,S = H ζ0 (M) and iterate: L P 1 (M S) and S H ζ (M L). Incoherence of L L = u (u ) and u µ n and L µ2 n. Solution for handling large S First threshold M before rank-1 projection. Ensures large entries of S are identified. Choose threshold ζ 0 = 4µ2 n.
39 Rank-1 Analysis Contd. Non-convex method (AltProj) L (0) = 0,S (0) = H ζ0 (M), L (t+1) P 1 (M S (t) ),S (t+1) H ζ (M L (t+1) ). L S S (0) M L (1) H ζ1 H ζ2 To analyze progress, track E (t+1) := S S (t+1) P 1
40 Rank-1 Analysis Contd. One iteration of AltProj L (0) = 0,S (0) = H ζ0 (M), L (1) P 1 (M S (0) ), S (1) H ζ (M L (1) ). Analyze E (1) := S S (1) Thresholding is element-wise operation: require L (1) L. In general, no special bound for L (1) L. Exploit sparsity of S and incoherence of L?
41 Rank-1 Analysis Contd. L (1) = uu = P 1 (M S (0) ) and E (0) = S S (0). Fixed point equation for eigenvectors (M S (0) )u = λu
42 Rank-1 Analysis Contd. L (1) = uu = P 1 (M S (0) ) and E (0) = S S (0). Fixed point equation for eigenvectors (M S (0) )u = λu ( u,u u +(S S (0) )u = λu or u = λ u,u I E(0) λ ) 1u Taylor Series u = λ u,u I + p 1 ( E (0) λ ) p u
43 Rank-1 Analysis Contd. L (1) = uu = P 1 (M S (0) ) and E (0) = S S (0). Fixed point equation for eigenvectors (M S (0) )u = λu ( u,u u +(S S (0) )u = λu or u = λ u,u I E(0) λ ) 1u Taylor Series u = λ u,u I + p 1 ( E (0) λ ) p u E (0) is sparse: supp(e (0) ) supp(s ).
44 Rank-1 Analysis Contd. L (1) = uu = P 1 (M S (0) ) and E (0) = S S (0). Fixed point equation for eigenvectors (M S (0) )u = λu ( u,u u +(S S (0) )u = λu or u = λ u,u I E(0) λ ) 1u Taylor Series u = λ u,u I + p 1 ( E (0) λ ) p u E (0) is sparse: supp(e (0) ) supp(s ). Exploiting sparsity: (E (0) ) p is the p th -hop adjacency matrix of E (0).
45 Rank-1 Analysis Contd. L (1) = uu = P 1 (M S (0) ) and E (0) = S S (0). Fixed point equation for eigenvectors (M S (0) )u = λu ( u,u u +(S S (0) )u = λu or u = λ u,u I E(0) λ ) 1u Taylor Series u = λ u,u I + p 1 ( E (0) λ ) p u E (0) is sparse: supp(e (0) ) supp(s ). Exploiting sparsity: (E (0) ) p is the p th -hop adjacency matrix of E (0). Counting walks in sparse graphs.
46 Rank-1 Analysis Contd. L (1) = uu = P 1 (M S (0) ) and E (0) = S S (0). Fixed point equation for eigenvectors (M S (0) )u = λu ( u,u u +(S S (0) )u = λu or u = λ u,u I E(0) λ ) 1u Taylor Series u = λ u,u I + p 1 ( E (0) λ ) p u E (0) is sparse: supp(e (0) ) supp(s ). Exploiting sparsity: (E (0) ) p is the p th -hop adjacency matrix of E (0). Counting walks in sparse graphs. In addition, u is incoherent: u < µ n.
47 Rank-1 Analysis Contd. u = λ u,u I + p 1 ( E (0) λ ) p u E (0) is sparse (each row/column is d sparse) and u is µ-incoherent.
48 Rank-1 Analysis Contd. u = λ u,u I + p 1 ( E (0) λ ) p u E (0) is sparse (each row/column is d sparse) and u is µ-incoherent. We show: (E (0) ) p u µ n (d E (0) ) p.
49 Rank-1 Analysis Contd. u = λ u,u I + p 1 ( E (0) λ ) p u E (0) is sparse (each row/column is d sparse) and u is µ-incoherent. We show: (E (0) ) p u µ n (d E (0) ) p. Convergence when terms are < 1, i.e. d E (0) < 1.
50 Rank-1 Analysis Contd. u = λ u,u I + p 1 ( E (0) λ ) p u E (0) is sparse (each row/column is d sparse) and u is µ-incoherent. We show: (E (0) ) p u µ n (d E (0) ) p. Convergence when terms are < 1, i.e. d E (0) < 1. Recall E (0) < 4µ2 n due to thresholding.
51 Rank-1 Analysis Contd. u = λ u,u I + p 1 ( E (0) λ ) p u E (0) is sparse (each row/column is d sparse) and u is µ-incoherent. We show: (E (0) ) p u µ n (d E (0) ) p. Convergence when terms are < 1, i.e. d E (0) < 1. Recall E (0) < 4µ2 n due to thresholding. Require d < n. Can tolerate O(n) corruptions! 4µ 2
52 Rank-1 Analysis Contd. u = λ u,u I + p 1 ( E (0) λ ) p u E (0) is sparse (each row/column is d sparse) and u is µ-incoherent. We show: (E (0) ) p u µ n (d E (0) ) p. Convergence when terms are < 1, i.e. d E (0) < 1. Recall E (0) < 4µ2 n due to thresholding. Require d < n. Can tolerate O(n) corruptions! 4µ 2 Contraction of error E (t) when degree d is bounded.
53 Extension to general rank: challenges
54 Extension to general rank: challenges A proposal for rank-r Non-convex method (AltProj) Init L (0) = 0,S (0) = H ζ0 (M), iterate: L (t+1) P r (M S (t) ), S (t+1) H ζ (M L (t+1) ).
55 Extension to general rank: challenges A proposal for rank-r Non-convex method (AltProj) Init L (0) = 0,S (0) = H ζ0 (M), iterate: L (t+1) P r (M S (t) ), S (t+1) H ζ (M L (t+1) ). Recall for rank-1 case Initial threshold controlled perturbation for rank-1 projection.
56 Extension to general rank: challenges A proposal for rank-r Non-convex method (AltProj) Init L (0) = 0,S (0) = H ζ0 (M), iterate: L (t+1) P r (M S (t) ), S (t+1) H ζ (M L (t+1) ). Recall for rank-1 case Initial threshold controlled perturbation for rank-1 projection. Perturbation analysis in general rank case Small λ min (L ): no recovery of lower eigenvectors.
57 Extension to general rank: challenges A proposal for rank-r Non-convex method (AltProj) Init L (0) = 0,S (0) = H ζ0 (M), iterate: L (t+1) P r (M S (t) ), S (t+1) H ζ (M L (t+1) ). Recall for rank-1 case Initial threshold controlled perturbation for rank-1 projection. Perturbation analysis in general rank case Small λ min (L ): no recovery of lower eigenvectors. Sparsity level depends on condition number λ max/λ min
58 Extension to general rank: challenges A proposal for rank-r Non-convex method (AltProj) Init L (0) = 0,S (0) = H ζ0 (M), iterate: L (t+1) P r (M S (t) ), S (t+1) H ζ (M L (t+1) ). Recall for rank-1 case Initial threshold controlled perturbation for rank-1 projection. Perturbation analysis in general rank case Small λ min (L ): no recovery of lower eigenvectors. Sparsity level depends on condition number λ max/λ min Guarantees without dependence on condition number?
59 Extension to general rank: challenges A proposal for rank-r Non-convex method (AltProj) Init L (0) = 0,S (0) = H ζ0 (M), iterate: L (t+1) P r (M S (t) ), S (t+1) H ζ (M L (t+1) ). Recall for rank-1 case Initial threshold controlled perturbation for rank-1 projection. Perturbation analysis in general rank case Small λ min (L ): no recovery of lower eigenvectors. Sparsity level depends on condition number λ max/λ min Guarantees without dependence on condition number? Lower eigenvectors subject to a large perturbation initially.
60 Extension to general rank: challenges A proposal for rank-r Non-convex method (AltProj) Init L (0) = 0,S (0) = H ζ0 (M), iterate: L (t+1) P r (M S (t) ), S (t+1) H ζ (M L (t+1) ). Recall for rank-1 case Initial threshold controlled perturbation for rank-1 projection. Perturbation analysis in general rank case Small λ min (L ): no recovery of lower eigenvectors. Sparsity level depends on condition number λ max/λ min Guarantees without dependence on condition number? Lower eigenvectors subject to a large perturbation initially. Reduce perturbation before recovering lower eigenvectors!
61 Improved Algorithm for General Rank Setting Stage-wise Projections Init L (0) = 0,S (0) = H ζ0 (M). For stage k = 1 to r, Iterate: L (t+1) P k (M S (t) ), S (t+1) H ζ (M L (t+1) ).
62 Improved Algorithm for General Rank Setting Stage-wise Projections Init L (0) = 0,S (0) = H ζ0 (M). For stage k = 1 to r, Iterate: L (t+1) P k (M S (t) ), S (t+1) H ζ (M L (t+1) ). L S S (0) M L (1) H ζ1 P 1 P 2 P r H ζ2
63 Summary of Results Low rank part: L = U Λ (V ) has rank r. Incoherence: U (i,:) 2, V (i,:) 2 µ r. n Sparse part: S has at most d non-zeros per row/column.
64 Summary of Results Low rank part: L = U Λ (V ) has rank r. Incoherence: U (i,:) 2, V (i,:) 2 µ r. n Sparse part: S has at most d non-zeros per row/column. Theorem: Guarantees for Stage-wise AltProj ( ) n Exact recovery of L,S when d = O µ 2 r
65 Summary of Results Low rank part: L = U Λ (V ) has rank r. Incoherence: U (i,:) 2, V (i,:) 2 µ r. n Sparse part: S has at most d non-zeros per row/column. Theorem: Guarantees for Stage-wise AltProj ( ) n Exact recovery of L,S when d = O µ 2 r Computational complexity: O ( r 2 n 2 log(1/ǫ) )
66 Summary of Results Low rank part: L = U Λ (V ) has rank r. Incoherence: U (i,:) 2, V (i,:) 2 µ r. n Sparse part: S has at most d non-zeros per row/column. Theorem: Guarantees for Stage-wise AltProj ( ) n Exact recovery of L,S when d = O µ 2 r Computational complexity: O ( r 2 n 2 log(1/ǫ) ) Comparison to convex method Same (deterministic) condition on d. Running time: O ( n 3 /ǫ )
67 Summary of Results Low rank part: L = U Λ (V ) has rank r. Incoherence: U (i,:) 2, V (i,:) 2 µ r. n Sparse part: S has at most d non-zeros per row/column. Theorem: Guarantees for Stage-wise AltProj ( ) n Exact recovery of L,S when d = O µ 2 r Computational complexity: O ( r 2 n 2 log(1/ǫ) ) Comparison to convex method Same (deterministic) condition on d. Running time: O ( n 3 /ǫ ) Best of both worlds: reduced computation with guarantees! Non-convex Robust PCA, P. Netrapalli, U.N. Niranjan, S. Sanghavi, A., P. Jain, NIPS 14.
68 Outline 1 Introduction 2 Analysis 3 Experiments 4 Robust Tensor PCA 5 Conclusion
69 Synthetic Results NcRPCA: Non-convex Robust PCA. IALM: Inexact augmented Lagrange multipliers. n = 2000, r = 5, µ = 1 n = 2000, r = 10, n α = 100 n = 2000, n α = 3 r, µ = 1 Time(s) 10 2 NcRPCA IALM Time(s) 10 2 NcRPCA IALM Time(s) 10 2 NcRPCA IALM n α µ r Max. Rank n = 2000, n α = 3 r, µ = 1 IALM r Max. Rank n = 2000, r = 10, n α = 100 IALM µ Rank n = 2000, r = 10, n α = 100 IALM Iterations
70 Real data: Foreground/background Separation Original Rank-10 PCA AltProj IALM
71 Real data: Foreground/background Separation AltProj IALM
72 Outline 1 Introduction 2 Analysis 3 Experiments 4 Robust Tensor PCA 5 Conclusion
73 Robust Tensor PCA vs.
74 Robust Tensor PCA vs. Robust Tensor Problem
75 Robust Tensor PCA vs. Robust Tensor Problem Applications: Robust Learning of Latent Variable Models. A., R. Ge, D. Hsu, S.M. Kakade and M. Telgarsky Tensor Decompositions for Learning Latent Variable Models, Preprint, Oct. 12.
76 Challenges and Preliminary Observations Convex methods T = L +S R n n n, L = i [r] a 3 i. No natural convex surrogate for tensor (CP) rank. Matricization loses the tensor structure!
77 Challenges and Preliminary Observations Convex methods T = L +S R n n n, L = i [r] a 3 i. No natural convex surrogate for tensor (CP) rank. Matricization loses the tensor structure! Non-Convex Heuristic: Extension of Matrix AltProj L (t+1) P r (T S (t) ),S (t+1) H ζ (T L (t+1) ).
78 Challenges and Preliminary Observations Convex methods T = L +S R n n n, L = i [r] a 3 i. No natural convex surrogate for tensor (CP) rank. Matricization loses the tensor structure! Non-Convex Heuristic: Extension of Matrix AltProj L (t+1) P r (T S (t) ),S (t+1) H ζ (T L (t+1) ). Challenges in Non-Convex Analysis P r for a general tensor is NP-hard!
79 Challenges and Preliminary Observations Convex methods T = L +S R n n n, L = i [r] a 3 i. No natural convex surrogate for tensor (CP) rank. Matricization loses the tensor structure! Non-Convex Heuristic: Extension of Matrix AltProj L (t+1) P r (T S (t) ),S (t+1) H ζ (T L (t+1) ). Challenges in Non-Convex Analysis P r for a general tensor is NP-hard! Can be well approximated in special cases, e.g. full rank factors.
80 Challenges and Preliminary Observations Convex methods T = L +S R n n n, L = i [r] a 3 i. No natural convex surrogate for tensor (CP) rank. Matricization loses the tensor structure! Non-Convex Heuristic: Extension of Matrix AltProj L (t+1) P r (T S (t) ),S (t+1) H ζ (T L (t+1) ). Challenges in Non-Convex Analysis P r for a general tensor is NP-hard! Can be well approximated in special cases, e.g. full rank factors. Guaranteed recovery possible!
81 Outline 1 Introduction 2 Analysis 3 Experiments 4 Robust Tensor PCA 5 Conclusion
82 Conclusion M L S Guaranteed Non-Convex Robust PCA Simple non-convex method for robust PCA. Alternating rank projections and thresholding. Estimates for low rank and sparse parts grown gradually. Guarantees match convex methods. Low computational complexity: scalable to large matrices. Possible to have both: guarantees and low computation!
83 Outlook Reduce computational complexity? Skip stages in rank projections? Tight bounds for incoherent row-column subspaces? Extendable to the tensor setting with tight scaling guarantees. Other problems where non-convex methods have guarantees? Csiszar s alternating minimization framework. (Laserre) hierarchy for convex methods: increasing complexity for harder problems. Analogous unified thinking for non-convex methods? Holy grail: A general framework for non-convex methods?
Provable Alternating Minimization Methods for Non-convex Optimization
Provable Alternating Minimization Methods for Non-convex Optimization Prateek Jain Microsoft Research, India Joint work with Praneeth Netrapalli, Sujay Sanghavi, Alekh Agarwal, Animashree Anandkumar, Rashish
Robust Principal Component Analysis
ELE 538B: Mathematics of High-Dimensional Data Robust Principal Component Analysis Yuxin Chen Princeton University, Fall 2018 Disentangling sparse and low-rank matrices Suppose we are given a matrix M
Tensor Methods for Feature Learning
Tensor Methods for Feature Learning Anima Anandkumar U.C. Irvine Feature Learning For Efficient Classification Find good transformations of input for improved classification Figures used attributed to
Dictionary Learning Using Tensor Methods
Dictionary Learning Using Tensor Methods Anima Anandkumar U.C. Irvine Joint work with Rong Ge, Majid Janzamin and Furong Huang. Feature learning as cornerstone of ML ML Practice Feature learning as cornerstone
Structured matrix factorizations. Example: Eigenfaces
Structured matrix factorizations Example: Eigenfaces An extremely large variety of interesting and important problems in machine learning can be formulated as: Given a matrix, find a matrix and a matrix
Analysis of Robust PCA via Local Incoherence
Analysis of Robust PCA via Local Incoherence Huishuai Zhang Department of EECS Syracuse University Syracuse, NY 3244 hzhan23@syr.edu Yi Zhou Department of EECS Syracuse University Syracuse, NY 3244 yzhou35@syr.edu
Fast Algorithms for Robust PCA via Gradient Descent
Fast Algorithms for Robust PCA via Gradient Descent Xinyang Yi Dohyung Park Yudong Chen Constantine Caramanis The University of Texas at Austin Cornell University yixy,dhpark,constantine}@utexas.edu yudong.chen@cornell.edu
Orthogonal tensor decomposition
Orthogonal tensor decomposition Daniel Hsu Columbia University Largely based on 2012 arxiv report Tensor decompositions for learning latent variable models, with Anandkumar, Ge, Kakade, and Telgarsky.
Sum-of-Squares Method, Tensor Decomposition, Dictionary Learning
Sum-of-Squares Method, Tensor Decomposition, Dictionary Learning David Steurer Cornell Approximation Algorithms and Hardness, Banff, August 2014 for many problems (e.g., all UG-hard ones): better guarantees
Donald Goldfarb IEOR Department Columbia University UCLA Mathematics Department Distinguished Lecture Series May 17 19, 2016
Optimization for Tensor Models Donald Goldfarb IEOR Department Columbia University UCLA Mathematics Department Distinguished Lecture Series May 17 19, 2016 1 Tensors Matrix Tensor: higher-order matrix
Recovering any low-rank matrix, provably
Recovering any low-rank matrix, provably Rachel Ward University of Texas at Austin October, 2014 Joint work with Yudong Chen (U.C. Berkeley), Srinadh Bhojanapalli and Sujay Sanghavi (U.T. Austin) Matrix
Lecture Notes 10: Matrix Factorization
Optimization-based data analysis Fall 207 Lecture Notes 0: Matrix Factorization Low-rank models. Rank- model Consider the problem of modeling a quantity y[i, j] that depends on two indices i and j. To
Linear dimensionality reduction for data analysis
Linear dimensionality reduction for data analysis Nicolas Gillis Joint work with Robert Luce, François Glineur, Stephen Vavasis, Robert Plemmons, Gabriella Casalino The setup Dimensionality reduction for
Combining Sparsity with Physically-Meaningful Constraints in Sparse Parameter Estimation
UIUC CSL Mar. 24 Combining Sparsity with Physically-Meaningful Constraints in Sparse Parameter Estimation Yuejie Chi Department of ECE and BMI Ohio State University Joint work with Yuxin Chen (Stanford).
Efficient Spectral Methods for Learning Mixture Models!
Efficient Spectral Methods for Learning Mixture Models Qingqing Huang 2016 February Laboratory for Information & Decision Systems Based on joint works with Munther Dahleh, Rong Ge, Sham Kakade, Greg Valiant.
Sparse Subspace Clustering
Sparse Subspace Clustering Based on Sparse Subspace Clustering: Algorithm, Theory, and Applications by Elhamifar and Vidal (2013) Alex Gutierrez CSCI 8314 March 2, 2017 Outline 1 Motivation and Background
arxiv: v2 [cs.it] 19 Sep 2016
![arxiv: v2 [cs.it] 19 Sep 2016](/thumbs/93/117985004.jpg "arxiv: v2 [cs.it] 19 Sep 2016")
Fast Algorithms for Robust PCA via Gradient Descent Xinyang Yi Dohyung Park Yudong Chen Constantine Caramanis The University of Texas at Austin Cornell University {yixy,dhpark,constantine}@utexas.edu yudong.chen@cornell.edu
Robust Component Analysis via HQ Minimization
Robust Component Analysis via HQ Minimization Ran He, Wei-shi Zheng and Liang Wang 0-08-6 Outline Overview Half-quadratic minimization principal component analysis Robust principal component analysis Robust
Identifiability and Learning of Topic Models: Tensor Decompositions under Structural Constraints
Identifiability and Learning of Topic Models: Tensor Decompositions under Structural Constraints Anima Anandkumar U.C. Irvine Joint work with Daniel Hsu, Majid Janzamin Adel Javanmard and Sham Kakade.
Solving Corrupted Quadratic Equations, Provably
Solving Corrupted Quadratic Equations, Provably Yuejie Chi London Workshop on Sparse Signal Processing September 206 Acknowledgement Joint work with Yuanxin Li (OSU), Huishuai Zhuang (Syracuse) and Yingbin
Spectral k-support Norm Regularization
Spectral k-support Norm Regularization Andrew McDonald Department of Computer Science, UCL (Joint work with Massimiliano Pontil and Dimitris Stamos) 25 March, 2015 1 / 19 Problem: Matrix Completion Goal:
Certifying the Global Optimality of Graph Cuts via Semidefinite Programming: A Theoretic Guarantee for Spectral Clustering
Certifying the Global Optimality of Graph Cuts via Semidefinite Programming: A Theoretic Guarantee for Spectral Clustering Shuyang Ling Courant Institute of Mathematical Sciences, NYU Aug 13, 2018 Joint
Sketching for Large-Scale Learning of Mixture Models
Sketching for Large-Scale Learning of Mixture Models Nicolas Keriven Université Rennes 1, Inria Rennes Bretagne-atlantique Adv. Rémi Gribonval Outline Introduction Practical Approach Results Theoretical
Robust PCA via Outlier Pursuit
Robust PCA via Outlier Pursuit Huan Xu Electrical and Computer Engineering University of Texas at Austin huan.xu@mail.utexas.edu Constantine Caramanis Electrical and Computer Engineering University of
Sparse and Low-Rank Matrix Decompositions
Forty-Seventh Annual Allerton Conference Allerton House, UIUC, Illinois, USA September 30 - October 2, 2009 Sparse and Low-Rank Matrix Decompositions Venkat Chandrasekaran, Sujay Sanghavi, Pablo A. Parrilo,
arxiv: v1 [cs.cv] 17 Nov 2015
![arxiv: v1 [cs.cv] 17 Nov 2015](/thumbs/91/106238795.jpg "arxiv: v1 [cs.cv] 17 Nov 2015")
Robust PCA via Nonconvex Rank Approximation Zhao Kang, Chong Peng, Qiang Cheng Department of Computer Science, Southern Illinois University, Carbondale, IL 6901, USA {zhao.kang, pchong, qcheng}@siu.edu
ECS289: Scalable Machine Learning
ECS289: Scalable Machine Learning Cho-Jui Hsieh UC Davis Oct 11, 2016 Paper presentations and final project proposal Send me the names of your group member (2 or 3 students) before October 15 (this Friday)
Some Recent Advances. in Non-convex Optimization. Purushottam Kar IIT KANPUR
Some Recent Advances in Non-convex Optimization Purushottam Kar IIT KANPUR Outline of the Talk Recap of Convex Optimization Why Non-convex Optimization? Non-convex Optimization: A Brief Introduction Robust
26 : Spectral GMs. Lecturer: Eric P. Xing Scribes: Guillermo A Cidre, Abelino Jimenez G.
10-708: Probabilistic Graphical Models, Spring 2015 26 : Spectral GMs Lecturer: Eric P. Xing Scribes: Guillermo A Cidre, Abelino Jimenez G. 1 Introduction A common task in machine learning is to work with
Robust Asymmetric Nonnegative Matrix Factorization
Robust Asymmetric Nonnegative Matrix Factorization Hyenkyun Woo Korea Institue for Advanced Study Math @UCLA Feb. 11. 2015 Joint work with Haesun Park Georgia Institute of Technology, H. Woo (KIAS) RANMF
2.3. Clustering or vector quantization 57
Multivariate Statistics non-negative matrix factorisation and sparse dictionary learning The PCA decomposition is by construction optimal solution to argmin A R n q,h R q p X AH 2 2 under constraint :
Collaborative Filtering Matrix Completion Alternating Least Squares
Case Study 4: Collaborative Filtering Collaborative Filtering Matrix Completion Alternating Least Squares Machine Learning for Big Data CSE547/STAT548, University of Washington Sham Kakade May 19, 2016
Research Statement Qing Qu
Qing Qu Research Statement 1/4 Qing Qu (qq2105@columbia.edu) Today we are living in an era of information explosion. As the sensors and sensing modalities proliferate, our world is inundated by unprecedented
Guaranteed Learning of Latent Variable Models through Spectral and Tensor Methods
Guaranteed Learning of Latent Variable Models through Spectral and Tensor Methods Anima Anandkumar U.C. Irvine Application 1: Clustering Basic operation of grouping data points. Hypothesis: each data point
Robust PCA via Outlier Pursuit
1 Robust PCA via Outlier Pursuit Huan Xu, Constantine Caramanis, Member, and Sujay Sanghavi, Member Abstract Singular Value Decomposition (and Principal Component Analysis) is one of the most widely used
Dense Error Correction for Low-Rank Matrices via Principal Component Pursuit
Dense Error Correction for Low-Rank Matrices via Principal Component Pursuit Arvind Ganesh, John Wright, Xiaodong Li, Emmanuel J. Candès, and Yi Ma, Microsoft Research Asia, Beijing, P.R.C Dept. of Electrical
SPARSE signal representations have gained popularity in recent
6958 IEEE TRANSACTIONS ON INFORMATION THEORY, VOL. 57, NO. 10, OCTOBER 2011 Blind Compressed Sensing Sivan Gleichman and Yonina C. Eldar, Senior Member, IEEE Abstract The fundamental principle underlying
The convex algebraic geometry of rank minimization
The convex algebraic geometry of rank minimization Pablo A. Parrilo Laboratory for Information and Decision Systems Massachusetts Institute of Technology International Symposium on Mathematical Programming
Matrix factorization models for patterns beyond blocks. Pauli Miettinen 18 February 2016
Matrix factorization models for patterns beyond blocks 18 February 2016 What does a matrix factorization do?? A = U V T 2 For SVD that s easy! 3 Inner-product interpretation Element (AB) ij is the inner
arxiv: v1 [math.oc] 11 Jun 2009
![arxiv: v1 [math.oc] 11 Jun 2009](/thumbs/75/71986197.jpg "arxiv: v1 [math.oc] 11 Jun 2009")
RANK-SPARSITY INCOHERENCE FOR MATRIX DECOMPOSITION VENKAT CHANDRASEKARAN, SUJAY SANGHAVI, PABLO A. PARRILO, S. WILLSKY AND ALAN arxiv:0906.2220v1 [math.oc] 11 Jun 2009 Abstract. Suppose we are given a
Recent Advances in Non-Convex Optimization and its Implications to Learning
Recent Advances in Non-Convex Optimization and its Implications to Learning Anima Anandkumar.. U.C. Irvine.. ICML 2016 Tutorial Optimization at the heart of Machine Learning Unsupervised Learning Most
Parallel Numerical Algorithms
Parallel Numerical Algorithms Chapter 6 Structured and Low Rank Matrices Section 6.3 Numerical Optimization Michael T. Heath and Edgar Solomonik Department of Computer Science University of Illinois at
CSC 576: Variants of Sparse Learning
CSC 576: Variants of Sparse Learning Ji Liu Department of Computer Science, University of Rochester October 27, 205 Introduction Our previous note basically suggests using l norm to enforce sparsity in
Machine Learning (CSE 446): Learning as Minimizing Loss; Least Squares
Machine Learning (CSE 446): Learning as Minimizing Loss; Least Squares Sham M Kakade c 2018 University of Washington cse446-staff@cs.washington.edu 1 / 13 Review 1 / 13 Alternate View of PCA: Minimizing
Scalable Subspace Clustering
Scalable Subspace Clustering René Vidal Center for Imaging Science, Laboratory for Computational Sensing and Robotics, Institute for Computational Medicine, Department of Biomedical Engineering, Johns
Sparse and Robust Optimization and Applications
Sparse and and Statistical Learning Workshop Les Houches, 2013 Robust Laurent El Ghaoui with Mert Pilanci, Anh Pham EECS Dept., UC Berkeley January 7, 2013 1 / 36 Outline Sparse Sparse Sparse Probability
Low Rank Matrix Completion Formulation and Algorithm
1 2 Low Rank Matrix Completion and Algorithm Jian Zhang Department of Computer Science, ETH Zurich zhangjianthu@gmail.com March 25, 2014 Movie Rating 1 2 Critic A 5 5 Critic B 6 5 Jian 9 8 Kind Guy B 9
PCA and Robust PCA for Modern Datasets: Theory, Algorithms, and Applications
PCA and Robust PCA for Modern Datasets: Theory, Algorithms, and Applications Namrata Vaswani Dept. of Electrical and Computer Engineering Iowa State University Web: http://www.ece.iastate.edu/~namrata
Machine Learning for Signal Processing Sparse and Overcomplete Representations
Machine Learning for Signal Processing Sparse and Overcomplete Representations Abelino Jimenez (slides from Bhiksha Raj and Sourish Chaudhuri) Oct 1, 217 1 So far Weights Data Basis Data Independent ICA
Statistical and Computational Phase Transitions in Planted Models
Statistical and Computational Phase Transitions in Planted Models Jiaming Xu Joint work with Yudong Chen (UC Berkeley) Acknowledgement: Prof. Bruce Hajek November 4, 203 Cluster/Community structure in
ECE 8201: Low-dimensional Signal Models for High-dimensional Data Analysis
ECE 8201: Low-dimensional Signal Models for High-dimensional Data Analysis Lecture 7: Matrix completion Yuejie Chi The Ohio State University Page 1 Reference Guaranteed Minimum-Rank Solutions of Linear
ECS289: Scalable Machine Learning
ECS289: Scalable Machine Learning Cho-Jui Hsieh UC Davis Oct 27, 2015 Outline One versus all/one versus one Ranking loss for multiclass/multilabel classification Scaling to millions of labels Multiclass
ECE G: Special Topics in Signal Processing: Sparsity, Structure, and Inference
ECE 18-898G: Special Topics in Signal Processing: Sparsity, Structure, and Inference Low-rank matrix recovery via nonconvex optimization Yuejie Chi Department of Electrical and Computer Engineering Spring
Nearly Optimal Robust Subspace Tracking
Praneeth Narayanamurthy 1 Namrata Vaswani 1 Abstract Robust subspace tracking (RST) can be simply understood as a dynamic (time-varying) extension of robust PCA. More precisely, it is the problem of tracking
GoDec: Randomized Low-rank & Sparse Matrix Decomposition in Noisy Case
ianyi Zhou IANYI.ZHOU@SUDEN.US.EDU.AU Dacheng ao DACHENG.AO@US.EDU.AU Centre for Quantum Computation & Intelligent Systems, FEI, University of echnology, Sydney, NSW 2007, Australia Abstract Low-rank and
Estimating Latent Variable Graphical Models with Moments and Likelihoods
Estimating Latent Variable Graphical Models with Moments and Likelihoods Arun Tejasvi Chaganty Percy Liang Stanford University June 18, 2014 Chaganty, Liang (Stanford University) Moments and Likelihoods
An ADMM Algorithm for Clustering Partially Observed Networks
An ADMM Algorithm for Clustering Partially Observed Networks Necdet Serhat Aybat Industrial Engineering Penn State University 2015 SIAM International Conference on Data Mining Vancouver, Canada Problem
Rapid, Robust, and Reliable Blind Deconvolution via Nonconvex Optimization
Rapid, Robust, and Reliable Blind Deconvolution via Nonconvex Optimization Shuyang Ling Department of Mathematics, UC Davis Oct.18th, 2016 Shuyang Ling (UC Davis) 16w5136, Oaxaca, Mexico Oct.18th, 2016
Learning Tractable Graphical Models: Latent Trees and Tree Mixtures
Learning Tractable Graphical Models: Latent Trees and Tree Mixtures Anima Anandkumar U.C. Irvine Joint work with Furong Huang, U.N. Niranjan, Daniel Hsu and Sham Kakade. High-Dimensional Graphical Modeling
Conditions for Robust Principal Component Analysis
Rose-Hulman Undergraduate Mathematics Journal Volume 12 Issue 2 Article 9 Conditions for Robust Principal Component Analysis Michael Hornstein Stanford University, mdhornstein@gmail.com Follow this and
Optimization for Compressed Sensing
Optimization for Compressed Sensing Robert J. Vanderbei 2014 March 21 Dept. of Industrial & Systems Engineering University of Florida http://www.princeton.edu/ rvdb Lasso Regression The problem is to solve
Machine Learning for Signal Processing Sparse and Overcomplete Representations. Bhiksha Raj (slides from Sourish Chaudhuri) Oct 22, 2013
Machine Learning for Signal Processing Sparse and Overcomplete Representations Bhiksha Raj (slides from Sourish Chaudhuri) Oct 22, 2013 1 Key Topics in this Lecture Basics Component-based representations
Finding a sparse vector in a subspace: Linear sparsity using alternating directions
Finding a sparse vector in a subspace: Linear sparsity using alternating directions Qing Qu, Ju Sun, and John Wright {qq05, js4038, jw966}@columbia.edu Dept. of Electrical Engineering, Columbia University,
Distributed Inexact Newton-type Pursuit for Non-convex Sparse Learning
Distributed Inexact Newton-type Pursuit for Non-convex Sparse Learning Bo Liu Department of Computer Science, Rutgers Univeristy Xiao-Tong Yuan BDAT Lab, Nanjing University of Information Science and Technology
An Introduction to Spectral Learning
An Introduction to Spectral Learning Hanxiao Liu November 8, 2013 Outline 1 Method of Moments 2 Learning topic models using spectral properties 3 Anchor words Preliminaries X 1,, X n p (x; θ), θ = (θ 1,
ECS289: Scalable Machine Learning
ECS289: Scalable Machine Learning Cho-Jui Hsieh UC Davis Oct 18, 2016 Outline One versus all/one versus one Ranking loss for multiclass/multilabel classification Scaling to millions of labels Multiclass
Notes on Latent Semantic Analysis
Notes on Latent Semantic Analysis Costas Boulis 1 Introduction One of the most fundamental problems of information retrieval (IR) is to find all documents (and nothing but those) that are semantically
Lecture 14: SVD, Power method, and Planted Graph problems (+ eigenvalues of random matrices) Lecturer: Sanjeev Arora
princeton univ. F 13 cos 521: Advanced Algorithm Design Lecture 14: SVD, Power method, and Planted Graph problems (+ eigenvalues of random matrices) Lecturer: Sanjeev Arora Scribe: Today we continue the
Spectral Clustering using Multilinear SVD
Spectral Clustering using Multilinear SVD Analysis, Approximations and Applications Debarghya Ghoshdastidar, Ambedkar Dukkipati Dept. of Computer Science & Automation Indian Institute of Science Clustering:
Sparsifying Transform Learning for Compressed Sensing MRI
Sparsifying Transform Learning for Compressed Sensing MRI Saiprasad Ravishankar and Yoram Bresler Department of Electrical and Computer Engineering and Coordinated Science Laborarory University of Illinois
Matrix completion: Fundamental limits and efficient algorithms. Sewoong Oh Stanford University
Matrix completion: Fundamental limits and efficient algorithms Sewoong Oh Stanford University 1 / 35 Low-rank matrix completion Low-rank Data Matrix Sparse Sampled Matrix Complete the matrix from small
Matrix-Tensor and Deep Learning in High Dimensional Data Analysis
Matrix-Tensor and Deep Learning in High Dimensional Data Analysis Tien D. Bui Department of Computer Science and Software Engineering Concordia University 14 th ICIAR Montréal July 5-7, 2017 Introduction
Optimal Value Function Methods in Numerical Optimization Level Set Methods
Optimal Value Function Methods in Numerical Optimization Level Set Methods James V Burke Mathematics, University of Washington, (jvburke@uw.edu) Joint work with Aravkin (UW), Drusvyatskiy (UW), Friedlander
SVD, Power method, and Planted Graph problems (+ eigenvalues of random matrices)
Chapter 14 SVD, Power method, and Planted Graph problems (+ eigenvalues of random matrices) Today we continue the topic of low-dimensional approximation to datasets and matrices. Last time we saw the singular
Robust PCA via Outlier Pursuit
1 Robust PCA via Outlier Pursuit Huan Xu, Constantine Caramanis, Member, and Sujay Sanghavi, Member Abstract Singular Value Decomposition (and Principal Component Analysis) is one of the most widely used
Algorithmic Foundations Of Data Sciences: Lectures explaining convex and non-convex optimization
Algorithmic Foundations Of Data Sciences: Lectures explaining convex and non-convex optimization INSTRUCTOR: CHANDRAJIT BAJAJ ( bajaj@cs.utexas.edu ) http://www.cs.utexas.edu/~bajaj Problems A. Sparse
Introduction to Machine Learning. PCA and Spectral Clustering. Introduction to Machine Learning, Slides: Eran Halperin
1 Introduction to Machine Learning PCA and Spectral Clustering Introduction to Machine Learning, 2013-14 Slides: Eran Halperin Singular Value Decomposition (SVD) The singular value decomposition (SVD)
Phase Retrieval from Local Measurements: Deterministic Measurement Constructions and Efficient Recovery Algorithms
Phase Retrieval from Local Measurements: Deterministic Measurement Constructions and Efficient Recovery Algorithms Aditya Viswanathan Department of Mathematics and Statistics adityavv@umich.edu www-personal.umich.edu/
EECS 275 Matrix Computation
EECS 275 Matrix Computation Ming-Hsuan Yang Electrical Engineering and Computer Science University of California at Merced Merced, CA 95344 http://faculty.ucmerced.edu/mhyang Lecture 22 1 / 21 Overview
ECE 598: Representation Learning: Algorithms and Models Fall 2017
ECE 598: Representation Learning: Algorithms and Models Fall 2017 Lecture 1: Tensor Methods in Machine Learning Lecturer: Pramod Viswanathan Scribe: Bharath V Raghavan, Oct 3, 2017 11 Introduction Tensors
Vector Space Models. wine_spectral.r
Vector Space Models 137 wine_spectral.r Latent Semantic Analysis Problem with words Even a small vocabulary as in wine example is challenging LSA Reduce number of columns of DTM by principal components
Semidefinite Programming Based Preconditioning for More Robust Near-Separable Nonnegative Matrix Factorization
Semidefinite Programming Based Preconditioning for More Robust Near-Separable Nonnegative Matrix Factorization Nicolas Gillis nicolas.gillis@umons.ac.be https://sites.google.com/site/nicolasgillis/ Department
Copyright by SIAM. Unauthorized reproduction of this article is prohibited.
SIAM J. OPTIM. Vol. 21, No. 2, pp. 572 596 2011 Society for Industrial and Applied Mathematics RANK-SPARSITY INCOHERENCE FOR MATRIX DECOMPOSITION * VENKAT CHANDRASEKARAN, SUJAY SANGHAVI, PABLO A. PARRILO,
Lecture 21: Spectral Learning for Graphical Models
10-708: Probabilistic Graphical Models 10-708, Spring 2016 Lecture 21: Spectral Learning for Graphical Models Lecturer: Eric P. Xing Scribes: Maruan Al-Shedivat, Wei-Cheng Chang, Frederick Liu 1 Motivation
Multi-Step Stochastic ADMM in High Dimensions: Applications to Sparse Optimization and Matrix Decomposition
Multi-Step Stochastic ADMM in High Dimensions: Applications to Sparse Optimization and Matrix Decomposition Hanie Sedghi Univ. of Southern California Los Angeles, CA 90089 hsedghi@usc.edu Anima Anandkumar
EE 381V: Large Scale Learning Spring Lecture 16 March 7
EE 381V: Large Scale Learning Spring 2013 Lecture 16 March 7 Lecturer: Caramanis & Sanghavi Scribe: Tianyang Bai 16.1 Topics Covered In this lecture, we introduced one method of matrix completion via SVD-based
LIMITATION OF LEARNING RANKINGS FROM PARTIAL INFORMATION. By Srikanth Jagabathula Devavrat Shah
00 AIM Workshop on Ranking LIMITATION OF LEARNING RANKINGS FROM PARTIAL INFORMATION By Srikanth Jagabathula Devavrat Shah Interest is in recovering distribution over the space of permutations over n elements
STATS 306B: Unsupervised Learning Spring Lecture 13 May 12
STATS 306B: Unsupervised Learning Spring 2014 Lecture 13 May 12 Lecturer: Lester Mackey Scribe: Jessy Hwang, Minzhe Wang 13.1 Canonical correlation analysis 13.1.1 Recap CCA is a linear dimensionality
Low-rank Matrix Completion with Noisy Observations: a Quantitative Comparison
Low-rank Matrix Completion with Noisy Observations: a Quantitative Comparison Raghunandan H. Keshavan, Andrea Montanari and Sewoong Oh Electrical Engineering and Statistics Department Stanford University,
Breaking the Limits of Subspace Inference
Breaking the Limits of Subspace Inference Claudia R. Solís-Lemus, Daniel L. Pimentel-Alarcón Emory University, Georgia State University Abstract Inferring low-dimensional subspaces that describe high-dimensional,
EE227 Project Report Robust Principal Component Analysis. Maximilian Balandat Walid Krichene Chi Pang Lam Ka Kit Lam
EE227 Project Report Robust Principal Component Analysis Maximilian Balandat Walid Krichene Chi Pang Lam Ka Kit Lam May 10, 2012 May 10, 2012 1 Introduction Over the past decade there has been an explosion
Tighter Low-rank Approximation via Sampling the Leveraged Element
Tighter Low-rank Approximation via Sampling the Leveraged Element Srinadh Bhojanapalli The University of Texas at Austin bsrinadh@utexas.edu Prateek Jain Microsoft Research, India prajain@microsoft.com
EUSIPCO
EUSIPCO 013 1569746769 SUBSET PURSUIT FOR ANALYSIS DICTIONARY LEARNING Ye Zhang 1,, Haolong Wang 1, Tenglong Yu 1, Wenwu Wang 1 Department of Electronic and Information Engineering, Nanchang University,
A Multi-Affine Model for Tensor Decomposition
Yiqing Yang UW Madison breakds@cs.wisc.edu A Multi-Affine Model for Tensor Decomposition Hongrui Jiang UW Madison hongrui@engr.wisc.edu Li Zhang UW Madison lizhang@cs.wisc.edu Chris J. Murphy UC Davis
Technical Report 110. Project: Streaming PCA with Many Missing Entries. Research Supervisor: Constantine Caramanis WNCG
Technical Report Project: Streaming PCA with Many Missing Entries Research Supervisor: Constantine Caramanis WNCG December 25 Data-Supported Transportation Operations & Planning Center D-STOP) A Tier USDOT
Provable Non-convex Phase Retrieval with Outliers: Median Truncated Wirtinger Flow
Provable Non-convex Phase Retrieval with Outliers: Median Truncated Wirtinger Flow Huishuai Zhang Department of EECS, Syracuse University, Syracuse, NY 3244 USA Yuejie Chi Department of ECE, The Ohio State
A Randomized Approach for Crowdsourcing in the Presence of Multiple Views
A Randomized Approach for Crowdsourcing in the Presence of Multiple Views Presenter: Yao Zhou joint work with: Jingrui He - 1 - Roadmap Motivation Proposed framework: M2VW Experimental results Conclusion
Detecting Sparse Structures in Data in Sub-Linear Time: A group testing approach
Detecting Sparse Structures in Data in Sub-Linear Time: A group testing approach Boaz Nadler The Weizmann Institute of Science Israel Joint works with Inbal Horev, Ronen Basri, Meirav Galun and Ery Arias-Castro
8.1 Concentration inequality for Gaussian random matrix (cont d)
MGMT 69: Topics in High-dimensional Data Analysis Falll 26 Lecture 8: Spectral clustering and Laplacian matrices Lecturer: Jiaming Xu Scribe: Hyun-Ju Oh and Taotao He, October 4, 26 Outline Concentration
Dimensionality Reduction and Principle Components Analysis
Dimensionality Reduction and Principle Components Analysis 1 Outline What is dimensionality reduction? Principle Components Analysis (PCA) Example (Bishop, ch 12) PCA vs linear regression PCA as a mixture