Sum-of-Squares Method, Tensor Decomposition, Dictionary Learning
|
|
- Jeffery Ward
- 5 years ago
- Views:
Transcription
1 Sum-of-Squares Method, Tensor Decomposition, Dictionary Learning David Steurer Cornell Approximation Algorithms and Hardness, Banff, August 2014
2 for many problems (e.g., all UG-hard ones): better guarantees stronger relaxations more complex proxy solutions??? possible relaxations (tractable problems) original problem (hard to solve directly) flows multicommodity, expander, electrical actual solutions metrics negative-type, simplex proxy solutions distributions Gaussian, random spanning tree
3 for many problems (e.g., all UG-hard ones): better guarantees stronger relaxations more complex proxy solutions??? possible relaxations (tractable problems) original problem (hard to solve directly) flows multicommodity, expander, electrical actual solutions metrics negative-type, simplex proxy solutions small price: need to use low-degree lens sum-of-squares allows us to avoid proxy solutions and focus on actual solutions distributions Gaussian, random spanning tree
4 example: max cut x i = 1 x i = 1 combinatorial viewpoint: polynomial viewpoint: given undirected graph G, bipartition vertex set to cut as many edges as possible given polynomial L G (x) = ij E G 1 4 x i x j 2, find maximum over x ±1 V(G) (hypercube) how to certify upper bound c on maximum? R R t decompose c L G as sum of squares of polynomials plus quadratic polynomial vanishing over hypercube α 1 x α n x n 2 1 n 2 -size semidefinite program
5 example: max cut x i = 1 x i = 1 combinatorial viewpoint: polynomial viewpoint: given undirected graph G, bipartition vertex set to cut as many edges as possible given polynomial L G (x) = ij E G 1 4 x i x j 2, find maximum over x ±1 V(G) (hypercube) how to certify upper bound c on maximum? R R t decompose c L G as sum of squares of polynomials plus quadratic polynomial vanishing over hypercube α 1 x α n x n 2 1 n 2 -size semidefinite program polynomial is sum of squares coefficient matrix is positive-semidefinite
6 example: max cut x i = 1 x i = 1 combinatorial viewpoint: polynomial viewpoint: given undirected graph G, bipartition vertex set to cut as many edges as possible given polynomial L G (x) = ij E G 1 4 x i x j 2, find maximum over x ±1 V(G) (hypercube) how to certify upper bound c on maximum? R R t decompose c L G as sum of squares of polynomials plus quadratic polynomial vanishing over hypercube α 1 x α n x n 2 1 n 2 -size semidefinite program Goemans-Williamson bound: either decomposition exists or max c ( best known approximation guarantee) spectral methods: add restriction α 1 = = α n largest Laplacian eigenvalue
7 example: max cut x i = 1 x i = 1 combinatorial viewpoint: polynomial viewpoint: how to certify upper bound c on maximum? given undirected graph G, bipartition vertex set to cut as many edges as possible given polynomial L G (x) = ij E G 1 4 x i x j 2, find maximum over x ±1 V(G) (hypercube) R R t decompose c L G as sum of squares of polynomials plus quadratic polynomial vanishing over hypercube degree-k α 1 x α n x n 2 1 n k n 2 -size semidefinite program degree-n bound is exact (interpolate c L G as degree-n polynomial over hypercube) does degree-n o(1) bound improve over GW in worst-case? (would refute UGC) for every* candidate graph construction, degree-16 bound improves over GW [Brandao-Barak-Harrow-Kelner-S.-Zhou]
8 multivariate polynomials P 1,, P m R x 1,, x n when is E unsatisfiable over R n? sum-of-squares (SOS) refutation of E system of equations E = P 1 = 0,, P m = 0 idea: derive 1 0 from E obviously unsatisfiable constraint always negative Q 1 P Q m P m + R R t 2 = 1 non-negative over E ( derived constraint ) intuitive proof system: many common inequalities have proofs in this form, e.g., Cauchy-Schwarz, Hölder, l p -triangle inequalities linear case: Gaussian elimination, Farkas lemma Real Nullstellensatz every polynomial system is either satisfiable over R n or SOS refutable [Artin, Krivine, Stengle] SOS method: n O k -time algorithm to find SOS refutation with degrees k if one exists (uses SDP) [Shor, Nesterov, Parrilo, Lasserre]
9 multivariate polynomials P 1,, P m R x 1,, x n system of equations E = P 1 = 0,, P m = 0 what if no deg.-d SOS refutation exists? degree-d SOS implication, E d P 0, P = Q 1 P Q m P m + R R t 2 with deg Q i P i, deg R i 2 d 1 linear functional E on deg.-d polynomials E1 = 1 P P 1P2 P m E EP 0 whenever E d P 0 no appreciable difference to deg.-d moments of some distribution over solutions to E P E d P 0 deg.-d pseudo-distribution for E
10 SOS implication: hypercube triangle inequality suppose: E = x 2 = 1, y 2 = 1, z 2 = 1 and P = x y 2 + y z 2 x z 2 claim: E 4 P 0 polynomial P as function on ±1 3 y P = 0 x z 0 x = y = z 8 x = y z x z therefore, P = 1 2 x + y 2 x z 2 + x 2 1 Q x + y 2 1 Q y + z 2 1 Q z square polynomial
11 SOS implication: univariate inequalities suppose: P univariate and P 0 over R claim: deg P P 0 proof by induction on deg P P α 0 choose: minimizer α of P α R then: P = P α + x α 2 P for some polynomial P with deg P < deg P squares sum of squares by ind. hyp. useful consequence deg P deg Q P Q x 1,, x n 0 for any polynomial Q R x 1,, x n concrete infinite family of global constraints (unclear how to get with other methods)
12 optimization (e.g., MAX CUT) maximize P 0 over P 1 = 0,, P m = 0 v-vs-v approximation: given: sat. system P 0 = v, P 1 = 0,, P m = 0 find: solution to P 0 = v, P 1 = 0,, P m = 0 [Barak-Kelner-S. 14] claim: SOS reduces approximation in time n O k to deg.-k combining subset / distr. x of solutions to P 0 = v, P 1 = 0,, P m = 0 represented by all degree-k moments of x, e.g., E {x} x 1 x k use only properties of moments / solutions with degree-k SOS proofs single solution x to P 0 v, P 1 = 0,, P m = 0 proof: deg.-k combiner cannot distinguish between actual distributions and deg.-k pseudo-distributions
13 uses only that Cov(x) is p.s.d. has deg.-2 SOS proof v E xx v = E v x 2 0 distribution of solutions {x} deg.-2 combiner for MAX CUT - sample Gaussian distribution ξ with same deg.-2 moments as {x} - output x = sign ξ analysis show: if ξ i, ξ j and x i, x j have same deg.-2 moments, then P x i x j P x i x j single solution x involves only 2 variables x i, x j has low-deg. SOS proof
14 dictionary learning (aka sparse coding) application: machine learning (feature extraction) neuroscience (model for visual cortex) data vectors linear transformation dictionary A = sparse vectors y 1 y T a 1 a m x 1 x T goal: given data vectors y 1,, y T, reconstruct A example: dictionary for natural images [Olshausen-Fields 96] previous works assume incoherence a 1,., a m unknown unit vectors in isotropic position x 1,, x t are i.i.d. samples from unknown nice distr. over sparse vectors (only small correlations between coord s) [Arora-Ge-Moitra, Agarwal-Anandkumar-Jain-Netrapalli-Tandon] previous methods (local search): only very sparse vectors, up to n non-zeros [Barak-Kelner-S. 14] sum-of-squares method: full sparsity range, up to constant fraction non-zeros (quasipolynomial-time for sparsity o(1); polynomial-time for n ε )
15 dictionary learning (aka sparse coding) application: machine learning (feature extraction) neuroscience (model for visual cortex) data vectors linear transformation dictionary A = sparse vectors example: dictionary for natural images [Olshausen-Fields 96] y 1 y T a 1 a m x 1 x T a 1,., a m unknown unit vectors in isotropic position x 1,, x t are i.i.d. samples from unknown nice distr. over sparse vectors (only small correlations between coord s) theorem: [Barak-Kelner-S. 14] suppose m = O n and correlations between coord s small enough then, O log n -SOS can recover set A {±a 1,, ±a m } in Hausdorff distance
16 theorem: [Barak-Kelner-S. 14] suppose m = O n and correlations between coord s small enough then, O log n -SOS can recover set A {±a 1,, ±a m } in Hausdorff distance ±a 1 ±a 2 ±a n 1 ±a m?????? 1. construct polynomial P 0 u = 1 T t y t, u 4 from data vectors can show: global optima of P 0 correspond to ±a 1,, ±a m (but no control over local optima of P 0 ) low-degree SOS proof 2. compute global optima of P 0 in general: NP-hard problem (even approximately) approach: use SOS method and degree-o log m combiner works because every solution set clustered around m points
17 theorem: [Barak-Kelner-S. 14] suppose m = O n and correlations between coord s small enough then, O log n -SOS can recover set A {±a 1,, ±a m } in Hausdorff distance ±a 1 ±a 2 ±a n 1 ±a m connection to robust tensor decomposition?????? M = t y t 4 close to i a i 4 in spectral norm claim: O log m -SOS finds components {±a i } 1. construct polynomial P 0 u = 1 T t y t, u 4 from data vectors can show: global optima of P 0 correspond to ±a 1,, ±a m (but no control over local optima of P 0 ) low-degree SOS proof 2. compute global optima of P 0 in general: NP-hard problem (even approximately) approach: use SOS method and degree-o log m combiner works because every solution set clustered around m points
18 given: 4-tensor M R n 4 that is ε-close to i a 4 i in spectral norm for orthonormal vectors a 1,, a n R n goal: find a 1,, a n polynomial system: E = M, x 4 = 1 ε x 2 2 = 1 distribution of solutions {x} to E single solution x
19 given: 4-tensor M R n 4 that is ε-close to i a 4 i in spectral norm for orthonormal vectors a 1,, a n R n goal: find a 1,, a n polynomial system: E = M, x 4 = 1 ε x 2 2 = 1 deg.-o(k) combiner for TENSOR DECOMP. - choose random unit vectors {w} ` - reweigh distribution {x} by w, x 2k so that P x w, x 2k P x - output top eigenvector of E xx ±a 1 ±a 2 ±a n 1 ±a m distribution of solutions {x} to E single solution x analysis has low-deg. SOS proof - solutions {x} clustered around ±a i - with probability 1/n O 1 w, a max w, a i 2 i>1 reweighing increases probability of a 1 -cluster by factor 2 k relative to rest for k = O log k, reweighted distr. concentrated along ±a 1
20 conclusions polynomial optimization: often easy when global optima unique (occurs naturally for recovery problems) unsupervised learning: higher-degree SOS gives better guarantees for recovering hidden structures low-degree combiner: general way to make proofs into algorithms thank you!
21
22
23
approximation algorithms I
SUM-OF-SQUARES method and approximation algorithms I David Steurer Cornell Cargese Workshop, 201 meta-task encoded as low-degree polynomial in R x example: f(x) = i,j n w ij x i x j 2 given: functions
More informationUnique Games Conjecture & Polynomial Optimization. David Steurer
Unique Games Conjecture & Polynomial Optimization David Steurer Newton Institute, Cambridge, July 2013 overview Proof Complexity Approximability Polynomial Optimization Quantum Information computational
More informationLower bounds on the size of semidefinite relaxations. David Steurer Cornell
Lower bounds on the size of semidefinite relaxations David Steurer Cornell James R. Lee Washington Prasad Raghavendra Berkeley Institute for Advanced Study, November 2015 overview of results unconditional
More informationSum-of-Squares and Spectral Algorithms
Sum-of-Squares and Spectral Algorithms Tselil Schramm June 23, 2017 Workshop on SoS @ STOC 2017 Spectral algorithms as a tool for analyzing SoS. SoS Semidefinite Programs Spectral Algorithms SoS suggests
More informationApproximation & Complexity
Summer school on semidefinite optimization Approximation & Complexity David Steurer Cornell University Part 1 September 6, 2012 Overview Part 1 Unique Games Conjecture & Basic SDP Part 2 SDP Hierarchies:
More informationSDP Relaxations for MAXCUT
SDP Relaxations for MAXCUT from Random Hyperplanes to Sum-of-Squares Certificates CATS @ UMD March 3, 2017 Ahmed Abdelkader MAXCUT SDP SOS March 3, 2017 1 / 27 Overview 1 MAXCUT, Hardness and UGC 2 LP
More informationTight Size-Degree Lower Bounds for Sums-of-Squares Proofs
Tight Size-Degree Lower Bounds for Sums-of-Squares Proofs Massimo Lauria KTH Royal Institute of Technology (Stockholm) 1 st Computational Complexity Conference, 015 Portland! Joint work with Jakob Nordström
More informationUnique Games and Small Set Expansion
Proof, beliefs, and algorithms through the lens of sum-of-squares 1 Unique Games and Small Set Expansion The Unique Games Conjecture (UGC) (Khot [2002]) states that for every ɛ > 0 there is some finite
More informationLecture 5. Max-cut, Expansion and Grothendieck s Inequality
CS369H: Hierarchies of Integer Programming Relaxations Spring 2016-2017 Lecture 5. Max-cut, Expansion and Grothendieck s Inequality Professor Moses Charikar Scribes: Kiran Shiragur Overview Here we derive
More informationMaximum cut and related problems
Proof, beliefs, and algorithms through the lens of sum-of-squares 1 Maximum cut and related problems Figure 1: The graph of Renato Paes Leme s friends on the social network Orkut, the partition to top
More informationLecture 4: Polynomial Optimization
CS369H: Hierarchies of Integer Programming Relaxations Spring 2016-2017 Lecture 4: Polynomial Optimization Professor Moses Charikar Scribes: Mona Azadkia 1 Introduction Non-negativity over the hypercube.
More informationHierarchies. 1. Lovasz-Schrijver (LS), LS+ 2. Sherali Adams 3. Lasserre 4. Mixed Hierarchy (recently used) Idea: P = conv(subset S of 0,1 n )
Hierarchies Today 1. Some more familiarity with Hierarchies 2. Examples of some basic upper and lower bounds 3. Survey of recent results (possible material for future talks) Hierarchies 1. Lovasz-Schrijver
More informationProvable Alternating Minimization Methods for Non-convex Optimization
Provable Alternating Minimization Methods for Non-convex Optimization Prateek Jain Microsoft Research, India Joint work with Praneeth Netrapalli, Sujay Sanghavi, Alekh Agarwal, Animashree Anandkumar, Rashish
More informationU.C. Berkeley CS294: Beyond Worst-Case Analysis Handout 12 Luca Trevisan October 3, 2017
U.C. Berkeley CS94: Beyond Worst-Case Analysis Handout 1 Luca Trevisan October 3, 017 Scribed by Maxim Rabinovich Lecture 1 In which we begin to prove that the SDP relaxation exactly recovers communities
More information6-1 The Positivstellensatz P. Parrilo and S. Lall, ECC
6-1 The Positivstellensatz P. Parrilo and S. Lall, ECC 2003 2003.09.02.10 6. The Positivstellensatz Basic semialgebraic sets Semialgebraic sets Tarski-Seidenberg and quantifier elimination Feasibility
More informationNon-convex Robust PCA: Provable Bounds
Non-convex Robust PCA: Provable Bounds Anima Anandkumar U.C. Irvine Joint work with Praneeth Netrapalli, U.N. Niranjan, Prateek Jain and Sujay Sanghavi. Learning with Big Data High Dimensional Regime Missing
More information8 Approximation Algorithms and Max-Cut
8 Approximation Algorithms and Max-Cut 8. The Max-Cut problem Unless the widely believed P N P conjecture is false, there is no polynomial algorithm that can solve all instances of an NP-hard problem.
More informationAgenda. Applications of semidefinite programming. 1 Control and system theory. 2 Combinatorial and nonconvex optimization
Agenda Applications of semidefinite programming 1 Control and system theory 2 Combinatorial and nonconvex optimization 3 Spectral estimation & super-resolution Control and system theory SDP in wide use
More informationConvex Optimization. (EE227A: UC Berkeley) Lecture 28. Suvrit Sra. (Algebra + Optimization) 02 May, 2013
Convex Optimization (EE227A: UC Berkeley) Lecture 28 (Algebra + Optimization) 02 May, 2013 Suvrit Sra Admin Poster presentation on 10th May mandatory HW, Midterm, Quiz to be reweighted Project final report
More informationComputational Lower Bounds for Statistical Estimation Problems
Computational Lower Bounds for Statistical Estimation Problems Ilias Diakonikolas (USC) (joint with Daniel Kane (UCSD) and Alistair Stewart (USC)) Workshop on Local Algorithms, MIT, June 2018 THIS TALK
More informationU.C. Berkeley CS294: Spectral Methods and Expanders Handout 11 Luca Trevisan February 29, 2016
U.C. Berkeley CS294: Spectral Methods and Expanders Handout Luca Trevisan February 29, 206 Lecture : ARV In which we introduce semi-definite programming and a semi-definite programming relaxation of sparsest
More informationConvex sets, conic matrix factorizations and conic rank lower bounds
Convex sets, conic matrix factorizations and conic rank lower bounds Pablo A. Parrilo Laboratory for Information and Decision Systems Electrical Engineering and Computer Science Massachusetts Institute
More informationIntroduction to Semidefinite Programming I: Basic properties a
Introduction to Semidefinite Programming I: Basic properties and variations on the Goemans-Williamson approximation algorithm for max-cut MFO seminar on Semidefinite Programming May 30, 2010 Semidefinite
More informationThe convex algebraic geometry of rank minimization
The convex algebraic geometry of rank minimization Pablo A. Parrilo Laboratory for Information and Decision Systems Massachusetts Institute of Technology International Symposium on Mathematical Programming
More informationCSC Linear Programming and Combinatorial Optimization Lecture 10: Semidefinite Programming
CSC2411 - Linear Programming and Combinatorial Optimization Lecture 10: Semidefinite Programming Notes taken by Mike Jamieson March 28, 2005 Summary: In this lecture, we introduce semidefinite programming
More informationLecture 12 : Graph Laplacians and Cheeger s Inequality
CPS290: Algorithmic Foundations of Data Science March 7, 2017 Lecture 12 : Graph Laplacians and Cheeger s Inequality Lecturer: Kamesh Munagala Scribe: Kamesh Munagala Graph Laplacian Maybe the most beautiful
More informationLecture 13: Spectral Graph Theory
CSE 521: Design and Analysis of Algorithms I Winter 2017 Lecture 13: Spectral Graph Theory Lecturer: Shayan Oveis Gharan 11/14/18 Disclaimer: These notes have not been subjected to the usual scrutiny reserved
More informationLecture 3: Semidefinite Programming
Lecture 3: Semidefinite Programming Lecture Outline Part I: Semidefinite programming, examples, canonical form, and duality Part II: Strong Duality Failure Examples Part III: Conditions for strong duality
More information2.1 Laplacian Variants
-3 MS&E 337: Spectral Graph heory and Algorithmic Applications Spring 2015 Lecturer: Prof. Amin Saberi Lecture 2-3: 4/7/2015 Scribe: Simon Anastasiadis and Nolan Skochdopole Disclaimer: hese notes have
More informationCertifying the Global Optimality of Graph Cuts via Semidefinite Programming: A Theoretic Guarantee for Spectral Clustering
Certifying the Global Optimality of Graph Cuts via Semidefinite Programming: A Theoretic Guarantee for Spectral Clustering Shuyang Ling Courant Institute of Mathematical Sciences, NYU Aug 13, 2018 Joint
More informationSemidefinite programming lifts and sparse sums-of-squares
1/15 Semidefinite programming lifts and sparse sums-of-squares Hamza Fawzi (MIT, LIDS) Joint work with James Saunderson (UW) and Pablo Parrilo (MIT) Cornell ORIE, October 2015 Linear optimization 2/15
More informationRobust and Optimal Control, Spring 2015
Robust and Optimal Control, Spring 2015 Instructor: Prof. Masayuki Fujita (S5-303B) G. Sum of Squares (SOS) G.1 SOS Program: SOS/PSD and SDP G.2 Duality, valid ineqalities and Cone G.3 Feasibility/Optimization
More informationRounding Sum-of-Squares Relaxations
Rounding Sum-of-Squares Relaxations Boaz Barak Jonathan Kelner David Steurer June 10, 2014 Abstract We present a general approach to rounding semidefinite programming relaxations obtained by the Sum-of-Squares
More informationMIT Algebraic techniques and semidefinite optimization February 14, Lecture 3
MI 6.97 Algebraic techniques and semidefinite optimization February 4, 6 Lecture 3 Lecturer: Pablo A. Parrilo Scribe: Pablo A. Parrilo In this lecture, we will discuss one of the most important applications
More informationAnalysis and synthesis: a complexity perspective
Analysis and synthesis: a complexity perspective Pablo A. Parrilo ETH ZürichZ control.ee.ethz.ch/~parrilo Outline System analysis/design Formal and informal methods SOS/SDP techniques and applications
More informationORIE 6334 Spectral Graph Theory September 8, Lecture 6. In order to do the first proof, we need to use the following fact.
ORIE 6334 Spectral Graph Theory September 8, 2016 Lecture 6 Lecturer: David P. Williamson Scribe: Faisal Alkaabneh 1 The Matrix-Tree Theorem In this lecture, we continue to see the usefulness of the graph
More informationGraph Partitioning Using Random Walks
Graph Partitioning Using Random Walks A Convex Optimization Perspective Lorenzo Orecchia Computer Science Why Spectral Algorithms for Graph Problems in practice? Simple to implement Can exploit very efficient
More informationOptimization over Nonnegative Polynomials: Algorithms and Applications. Amir Ali Ahmadi Princeton, ORFE
Optimization over Nonnegative Polynomials: Algorithms and Applications Amir Ali Ahmadi Princeton, ORFE INFORMS Optimization Society Conference (Tutorial Talk) Princeton University March 17, 2016 1 Optimization
More informationGraph Partitioning Algorithms and Laplacian Eigenvalues
Graph Partitioning Algorithms and Laplacian Eigenvalues Luca Trevisan Stanford Based on work with Tsz Chiu Kwok, Lap Chi Lau, James Lee, Yin Tat Lee, and Shayan Oveis Gharan spectral graph theory Use linear
More informationORIE 6334 Spectral Graph Theory November 22, Lecture 25
ORIE 64 Spectral Graph Theory November 22, 206 Lecture 25 Lecturer: David P. Williamson Scribe: Pu Yang In the remaining three lectures, we will cover a prominent result by Arora, Rao, and Vazirani for
More informationU.C. Berkeley CS294: Beyond Worst-Case Analysis Handout 8 Luca Trevisan September 19, 2017
U.C. Berkeley CS294: Beyond Worst-Case Analysis Handout 8 Luca Trevisan September 19, 2017 Scribed by Luowen Qian Lecture 8 In which we use spectral techniques to find certificates of unsatisfiability
More informationOn the efficient approximability of constraint satisfaction problems
On the efficient approximability of constraint satisfaction problems July 13, 2007 My world Max-CSP Efficient computation. P Polynomial time BPP Probabilistic Polynomial time (still efficient) NP Non-deterministic
More informationBasic Calculus Review
Basic Calculus Review Lorenzo Rosasco ISML Mod. 2 - Machine Learning Vector Spaces Functionals and Operators (Matrices) Vector Space A vector space is a set V with binary operations +: V V V and : R V
More informationIE 521 Convex Optimization
Lecture 14: and Applications 11th March 2019 Outline LP SOCP SDP LP SOCP SDP 1 / 21 Conic LP SOCP SDP Primal Conic Program: min c T x s.t. Ax K b (CP) : b T y s.t. A T y = c (CD) y K 0 Theorem. (Strong
More informationDictionary Learning Using Tensor Methods
Dictionary Learning Using Tensor Methods Anima Anandkumar U.C. Irvine Joint work with Rong Ge, Majid Janzamin and Furong Huang. Feature learning as cornerstone of ML ML Practice Feature learning as cornerstone
More informationLecture 14: Random Walks, Local Graph Clustering, Linear Programming
CSE 521: Design and Analysis of Algorithms I Winter 2017 Lecture 14: Random Walks, Local Graph Clustering, Linear Programming Lecturer: Shayan Oveis Gharan 3/01/17 Scribe: Laura Vonessen Disclaimer: These
More informationDonald Goldfarb IEOR Department Columbia University UCLA Mathematics Department Distinguished Lecture Series May 17 19, 2016
Optimization for Tensor Models Donald Goldfarb IEOR Department Columbia University UCLA Mathematics Department Distinguished Lecture Series May 17 19, 2016 1 Tensors Matrix Tensor: higher-order matrix
More informationFunctional Analysis Review
Outline 9.520: Statistical Learning Theory and Applications February 8, 2010 Outline 1 2 3 4 Vector Space Outline A vector space is a set V with binary operations +: V V V and : R V V such that for all
More informationDS-GA 1002 Lecture notes 0 Fall Linear Algebra. These notes provide a review of basic concepts in linear algebra.
DS-GA 1002 Lecture notes 0 Fall 2016 Linear Algebra These notes provide a review of basic concepts in linear algebra. 1 Vector spaces You are no doubt familiar with vectors in R 2 or R 3, i.e. [ ] 1.1
More informationCS 229r: Algorithms for Big Data Fall Lecture 19 Nov 5
CS 229r: Algorithms for Big Data Fall 215 Prof. Jelani Nelson Lecture 19 Nov 5 Scribe: Abdul Wasay 1 Overview In the last lecture, we started discussing the problem of compressed sensing where we are given
More informationSemidefinite Programming Basics and Applications
Semidefinite Programming Basics and Applications Ray Pörn, principal lecturer Åbo Akademi University Novia University of Applied Sciences Content What is semidefinite programming (SDP)? How to represent
More informationsublinear time low-rank approximation of positive semidefinite matrices Cameron Musco (MIT) and David P. Woodru (CMU)
sublinear time low-rank approximation of positive semidefinite matrices Cameron Musco (MIT) and David P. Woodru (CMU) 0 overview Our Contributions: 1 overview Our Contributions: A near optimal low-rank
More information5 Compact linear operators
5 Compact linear operators One of the most important results of Linear Algebra is that for every selfadjoint linear map A on a finite-dimensional space, there exists a basis consisting of eigenvectors.
More informationLectures 2 3 : Wigner s semicircle law
Fall 009 MATH 833 Random Matrices B. Való Lectures 3 : Wigner s semicircle law Notes prepared by: M. Koyama As we set up last wee, let M n = [X ij ] n i,j= be a symmetric n n matrix with Random entries
More informationGraph Sparsification I : Effective Resistance Sampling
Graph Sparsification I : Effective Resistance Sampling Nikhil Srivastava Microsoft Research India Simons Institute, August 26 2014 Graphs G G=(V,E,w) undirected V = n w: E R + Sparsification Approximate
More informationlinear programming and approximate constraint satisfaction
linear programming and approximate constraint satisfaction Siu On Chan MSR New England James R. Lee University of Washington Prasad Raghavendra UC Berkeley David Steurer Cornell results MAIN THEOREM: Any
More informationOrthogonal tensor decomposition
Orthogonal tensor decomposition Daniel Hsu Columbia University Largely based on 2012 arxiv report Tensor decompositions for learning latent variable models, with Anandkumar, Ge, Kakade, and Telgarsky.
More informationInverse Power Method for Non-linear Eigenproblems
Inverse Power Method for Non-linear Eigenproblems Matthias Hein and Thomas Bühler Anubhav Dwivedi Department of Aerospace Engineering & Mechanics 7th March, 2017 1 / 30 OUTLINE Motivation Non-Linear Eigenproblems
More informationLecture 17 (Nov 3, 2011 ): Approximation via rounding SDP: Max-Cut
CMPUT 675: Approximation Algorithms Fall 011 Lecture 17 (Nov 3, 011 ): Approximation via rounding SDP: Max-Cut Lecturer: Mohammad R. Salavatipour Scribe: based on older notes 17.1 Approximation Algorithm
More informationSparse and Low-Rank Matrix Decompositions
Forty-Seventh Annual Allerton Conference Allerton House, UIUC, Illinois, USA September 30 - October 2, 2009 Sparse and Low-Rank Matrix Decompositions Venkat Chandrasekaran, Sujay Sanghavi, Pablo A. Parrilo,
More informationLecture 10: October 27, 2016
Mathematical Toolkit Autumn 206 Lecturer: Madhur Tulsiani Lecture 0: October 27, 206 The conjugate gradient method In the last lecture we saw the steepest descent or gradient descent method for finding
More informationarxiv: v1 [math.oc] 26 Sep 2015
arxiv:1509.08021v1 [math.oc] 26 Sep 2015 Degeneracy in Maximal Clique Decomposition for Semidefinite Programs Arvind U. Raghunathan and Andrew V. Knyazev Mitsubishi Electric Research Laboratories 201 Broadway,
More informationLectures 2 3 : Wigner s semicircle law
Fall 009 MATH 833 Random Matrices B. Való Lectures 3 : Wigner s semicircle law Notes prepared by: M. Koyama As we set up last wee, let M n = [X ij ] n i,j=1 be a symmetric n n matrix with Random entries
More informationGrothendieck s Inequality
Grothendieck s Inequality Leqi Zhu 1 Introduction Let A = (A ij ) R m n be an m n matrix. Then A defines a linear operator between normed spaces (R m, p ) and (R n, q ), for 1 p, q. The (p q)-norm of A
More informationCS675: Convex and Combinatorial Optimization Fall 2016 Combinatorial Problems as Linear and Convex Programs. Instructor: Shaddin Dughmi
CS675: Convex and Combinatorial Optimization Fall 2016 Combinatorial Problems as Linear and Convex Programs Instructor: Shaddin Dughmi Outline 1 Introduction 2 Shortest Path 3 Algorithms for Single-Source
More informationConvex and Semidefinite Programming for Approximation
Convex and Semidefinite Programming for Approximation We have seen linear programming based methods to solve NP-hard problems. One perspective on this is that linear programming is a meta-method since
More informationLecture 9: Low Rank Approximation
CSE 521: Design and Analysis of Algorithms I Fall 2018 Lecture 9: Low Rank Approximation Lecturer: Shayan Oveis Gharan February 8th Scribe: Jun Qi Disclaimer: These notes have not been subjected to the
More informationPCA, Kernel PCA, ICA
PCA, Kernel PCA, ICA Learning Representations. Dimensionality Reduction. Maria-Florina Balcan 04/08/2015 Big & High-Dimensional Data High-Dimensions = Lot of Features Document classification Features per
More informationSemidefinite Programming
Semidefinite Programming Notes by Bernd Sturmfels for the lecture on June 26, 208, in the IMPRS Ringvorlesung Introduction to Nonlinear Algebra The transition from linear algebra to nonlinear algebra has
More informationSubsampling Semidefinite Programs and Max-Cut on the Sphere
Electronic Colloquium on Computational Complexity, Report No. 129 (2009) Subsampling Semidefinite Programs and Max-Cut on the Sphere Boaz Barak Moritz Hardt Thomas Holenstein David Steurer November 29,
More informationBEYOND MATRIX COMPLETION
BEYOND MATRIX COMPLETION ANKUR MOITRA MASSACHUSETTS INSTITUTE OF TECHNOLOGY Based on joint work with Boaz Barak (MSR) Part I: RecommendaIon systems and parially observed matrices THE NETFLIX PROBLEM movies
More informationECE 8201: Low-dimensional Signal Models for High-dimensional Data Analysis
ECE 8201: Low-dimensional Signal Models for High-dimensional Data Analysis Lecture 7: Matrix completion Yuejie Chi The Ohio State University Page 1 Reference Guaranteed Minimum-Rank Solutions of Linear
More information1 Regression with High Dimensional Data
6.883 Learning with Combinatorial Structure ote for Lecture 11 Instructor: Prof. Stefanie Jegelka Scribe: Xuhong Zhang 1 Regression with High Dimensional Data Consider the following regression problem:
More informationThe moment-lp and moment-sos approaches
The moment-lp and moment-sos approaches LAAS-CNRS and Institute of Mathematics, Toulouse, France CIRM, November 2013 Semidefinite Programming Why polynomial optimization? LP- and SDP- CERTIFICATES of POSITIVITY
More informationLecture 12: Introduction to Spectral Graph Theory, Cheeger s inequality
CSE 521: Design and Analysis of Algorithms I Spring 2016 Lecture 12: Introduction to Spectral Graph Theory, Cheeger s inequality Lecturer: Shayan Oveis Gharan May 4th Scribe: Gabriel Cadamuro Disclaimer:
More informationBEYOND MATRIX COMPLETION
BEYOND MATRIX COMPLETION ANKUR MOITRA MASSACHUSETTS INSTITUTE OF TECHNOLOGY Based on joint work with Boaz Barak (Harvard) Part I: RecommendaJon systems and parjally observed matrices THE NETFLIX PROBLEM
More informationIntroduction to Machine Learning. PCA and Spectral Clustering. Introduction to Machine Learning, Slides: Eran Halperin
1 Introduction to Machine Learning PCA and Spectral Clustering Introduction to Machine Learning, 2013-14 Slides: Eran Halperin Singular Value Decomposition (SVD) The singular value decomposition (SVD)
More informationAn 0.5-Approximation Algorithm for MAX DICUT with Given Sizes of Parts
An 0.5-Approximation Algorithm for MAX DICUT with Given Sizes of Parts Alexander Ageev Refael Hassin Maxim Sviridenko Abstract Given a directed graph G and an edge weight function w : E(G) R +, themaximumdirectedcutproblem(max
More informationPartitioning Algorithms that Combine Spectral and Flow Methods
CS369M: Algorithms for Modern Massive Data Set Analysis Lecture 15-11/11/2009 Partitioning Algorithms that Combine Spectral and Flow Methods Lecturer: Michael Mahoney Scribes: Kshipra Bhawalkar and Deyan
More informationLecture 20: Goemans-Williamson MAXCUT Approximation Algorithm. 2 Goemans-Williamson Approximation Algorithm for MAXCUT
CS 80: Introduction to Complexity Theory 0/03/03 Lecture 20: Goemans-Williamson MAXCUT Approximation Algorithm Instructor: Jin-Yi Cai Scribe: Christopher Hudzik, Sarah Knoop Overview First, we outline
More informationReductions Between Expansion Problems
Reductions Between Expansion Problems Prasad Raghavendra David Steurer Madhur Tulsiani November 11, 2010 Abstract The Small-Set Expansion Hypothesis (Raghavendra, Steurer, STOC 2010) is a natural hardness
More informationNear-Optimal Algorithms for Maximum Constraint Satisfaction Problems
Near-Optimal Algorithms for Maximum Constraint Satisfaction Problems Moses Charikar Konstantin Makarychev Yury Makarychev Princeton University Abstract In this paper we present approximation algorithms
More informationNon-Convex Optimization via Real Algebraic Geometry
Non-Convex Optimization via Real Algebraic Geometry Constantine Caramanis Massachusetts Institute of Technology November 29, 2001 The following paper represents the material from a collection of different
More informationTractable Upper Bounds on the Restricted Isometry Constant
Tractable Upper Bounds on the Restricted Isometry Constant Alex d Aspremont, Francis Bach, Laurent El Ghaoui Princeton University, École Normale Supérieure, U.C. Berkeley. Support from NSF, DHS and Google.
More informationSignal Recovery from Permuted Observations
EE381V Course Project Signal Recovery from Permuted Observations 1 Problem Shanshan Wu (sw33323) May 8th, 2015 We start with the following problem: let s R n be an unknown n-dimensional real-valued signal,
More informationA better approximation ratio for the Vertex Cover problem
A better approximation ratio for the Vertex Cover problem George Karakostas Dept. of Computing and Software McMaster University October 5, 004 Abstract We reduce the approximation factor for Vertex Cover
More informationComplexity of 10 Decision Problems in Continuous Time Dynamical Systems. Amir Ali Ahmadi IBM Watson Research Center
Complexity of 10 Decision Problems in Continuous Time Dynamical Systems Amir Ali Ahmadi IBM Watson Research Center Anirudha Majumdar MIT Russ Tedrake MIT 1 Given a polynomial vector field: Decide if the
More informationModeling with semidefinite and copositive matrices
Modeling with semidefinite and copositive matrices Franz Rendl http://www.math.uni-klu.ac.at Alpen-Adria-Universität Klagenfurt Austria F. Rendl, Singapore workshop 2006 p.1/24 Overview Node and Edge relaxations
More informationTHE HIDDEN CONVEXITY OF SPECTRAL CLUSTERING
THE HIDDEN CONVEXITY OF SPECTRAL CLUSTERING Luis Rademacher, Ohio State University, Computer Science and Engineering. Joint work with Mikhail Belkin and James Voss This talk A new approach to multi-way
More informationSpectral Graph Theory Lecture 2. The Laplacian. Daniel A. Spielman September 4, x T M x. ψ i = arg min
Spectral Graph Theory Lecture 2 The Laplacian Daniel A. Spielman September 4, 2015 Disclaimer These notes are not necessarily an accurate representation of what happened in class. The notes written before
More informationAcyclic Semidefinite Approximations of Quadratically Constrained Quadratic Programs
2015 American Control Conference Palmer House Hilton July 1-3, 2015. Chicago, IL, USA Acyclic Semidefinite Approximations of Quadratically Constrained Quadratic Programs Raphael Louca and Eilyan Bitar
More informationECE 598: Representation Learning: Algorithms and Models Fall 2017
ECE 598: Representation Learning: Algorithms and Models Fall 2017 Lecture 1: Tensor Methods in Machine Learning Lecturer: Pramod Viswanathan Scribe: Bharath V Raghavan, Oct 3, 2017 11 Introduction Tensors
More informationHow hard is it to find a good solution?
How hard is it to find a good solution? Simons Institute Open Lecture November 4, 2013 Research Area: Complexity Theory Given a computational problem, find an efficient algorithm that solves it. Goal of
More informationOptimization methods
Lecture notes 3 February 8, 016 1 Introduction Optimization methods In these notes we provide an overview of a selection of optimization methods. We focus on methods which rely on first-order information,
More informationTensor Methods for Feature Learning
Tensor Methods for Feature Learning Anima Anandkumar U.C. Irvine Feature Learning For Efficient Classification Find good transformations of input for improved classification Figures used attributed to
More informationORIE 6334 Spectral Graph Theory December 1, Lecture 27 Remix
ORIE 6334 Spectral Graph Theory December, 06 Lecturer: David P. Williamson Lecture 7 Remix Scribe: Qinru Shi Note: This is an altered version of the lecture I actually gave, which followed the structure
More informationReconstruction from Anisotropic Random Measurements
Reconstruction from Anisotropic Random Measurements Mark Rudelson and Shuheng Zhou The University of Michigan, Ann Arbor Coding, Complexity, and Sparsity Workshop, 013 Ann Arbor, Michigan August 7, 013
More informationStructured matrix factorizations. Example: Eigenfaces
Structured matrix factorizations Example: Eigenfaces An extremely large variety of interesting and important problems in machine learning can be formulated as: Given a matrix, find a matrix and a matrix
More informationLecture 2: November 9
Semidefinite programming and computational aspects of entanglement IHP Fall 017 Lecturer: Aram Harrow Lecture : November 9 Scribe: Anand (Notes available at http://webmitedu/aram/www/teaching/sdphtml)
More informationSupplementary lecture notes on linear programming. We will present an algorithm to solve linear programs of the form. maximize.
Cornell University, Fall 2016 Supplementary lecture notes on linear programming CS 6820: Algorithms 26 Sep 28 Sep 1 The Simplex Method We will present an algorithm to solve linear programs of the form
More information