approximation algorithms I

Transcription

1 SUM-OF-SQUARES method and approximation algorithms I David Steurer Cornell Cargese Workshop, 201

2 meta-task encoded as low-degree polynomial in R x example: f(x) = i,j n w ij x i x j 2 given: functions f 1,, f m : ±1 n R find: solution x ±1 n to f 1 = 0,, f m = 0 Laplacian L G = 1 E G ij E G 1 x 2 i x j examples: combinatorial optimization problem on graph G MAX CUT: MAX BISECTION: L G = 1 ε over ±1 n L G = 1 ε, i x i = 0 over ±1 n where 1 ε is guess for optimum value goal: develop SDP-based algorithms with provable guarantees in terms of complexity and approximation ( on the edge intractability need strongest possible relaxations)

3 meta-task given: functions f 1,, f m : ±1 n R find: solution x ±1 n to f 1 = 0,, f m = 0 goal: develop SDP-based algorithms with provable guarantees in terms of complexity and approximation price of convexity: individual solutions distributions over solutions price of tractability: can only enforce efficiently checkable knowledge about solutions distributions over solutions individual solutions pseudo-distributions over solutions (consistent with efficiently checkable knowledge)

4 distribution D over ±1 n examples uniform distribution: D = 2 n fixed 2-bit parity: D x = (1 + x 1 x 2 )/2 n function D: ±1 n R non-negativity: D x 0 for all x ±1 n # function values is exponential need careful representation normalization: x ±1 D x = 1 # independent inequalities is exponential not efficiently checkable distribution D satisfies f 1 = 0,, f m = 0 for some f i : ±1 n R E D f f m 2 = 0 (equivalently: P D i. f i 0 = 0) convex: D, D satisfy conditions D + D /2 satisfies conditions examples fixed 2-bit parity distribution satisfies x 1 x 2 = 1 uniform distribution does not satisfy f = 0 for any f 0

5 deg.-d pseudo-distribution D distribution D over ±1 n convenient notation: E D f x D x f x pseudo-expectation of f under D pseudo- function D: ±1 n R non-negativity: D x 0 for all x ±1 n normalization: x ±1 D x = 1 x ±1 n D x f x 2 0 for every deg.-d/2 polynomial f distribution D satisfies f 1 = 0,, f m = 0 for some f i : ±1 n R E D E D f f m 2 = 0 (equivalently: P D i. f i 0 = 0) deg.-2n pseudo-distributions are actual distributions 2 (point-indicators 1 x have deg. n D x = E D 1 x 0)

6 deg.-d pseudo-distr. D: ±1 n R notation: E D f x D x f x, pseudo-expectation of f under D non-negativity: E D f 2 0 for every deg.-d/2 poly. f normalization: E D 1 = 1 pseudo-distr. D satisfies f 1 = 0,, f m = 0 for some f i : ±1 n R E D f f 2 m = 0 (equivalently: E D f i g = 0 whenever deg g d deg f i )

7 deg.-d pseudo-distr. D: ±1 n R notation: E D f x D x f x, pseudo-expectation of f under D non-negativity: E D f 2 0 for every deg.-d/2 poly. f normalization: E D 1 = 1 pseudo-distr. D satisfies f 1 = 0,, f m = 0 for some f i : ±1 n R E D f f 2 m = 0 (equivalently: E D f i g = 0 whenever deg g d deg f i ) claim: can compute such D in time n O(d) if it exists (otherwise, certify that no solution to original problem exists) [Shor, Parrilo, Lasserre] (can assume D is deg.-d polynomial separation problem min f E D f 2 is n d - dim. eigenvalue prob. n O(d) -time via grad. descent / ellipsoid method)

8 deg.-d pseudo-distr. D: ±1 n R notation: E D f x D x f x, pseudo-expectation of f under D non-negativity: E D f 2 0 for every deg.-d/2 poly. f normalization: E D 1 = 1 pseudo-distr. D satisfies f 1 = 0,, f m = 0 for some f i : ±1 n R E D f f m 2 = 0 (equivalently: E D f i g = 0 whenever deg g d deg f i ) surprising property: E D f 0 for many* low-degree polynomials f such that f 0 follows from f 1 = 0,, f m = 0 by explicit proof soon: examples of such properties and how to exploit them

9 deg.-d pseudo-distr. D: ±1 n R notation: E D f x D x f x, pseudo-expectation of f under D non-negativity: E D f 2 0 for every deg.-d/2 poly. f normalization: E D 1 = 1 pseudo-distr. D satisfies f 1 = 0,, f m = 0 for some f i : ±1 n R E D f f 2 m = 0 (equivalently: n o(d) E D -time f i g algorithms = 0 whenever cannot* deg g distinguish d deg f i ) between deg.-d pseudo-distributions and surprising property: E D f 0 for many* deg.-d low-degree part of actual polynomials distr. s f such that f 0 follows from f 1 = 0,, f m = 0 by explicit proof soon: examples of such properties and how to exploit them deg.-d part of actual distr. over optimal solutions pseudo-distr. over optimal solutions efficient algorithm approximate solution (to original problem) emerging algorithm-design paradigm: analyze algorithm pretending that underlying actual distribution exists; verify only afterwards that low-deg. pseudo-distr. s satisfy required properties

10 dual view (sum-of-squares proof system) either deg.-d pseudo-distribution D over ±1 n satisfying f 1 = 0,, f m = 0 or g 1,, g m and h 1,, h k such that i f i g i + j h 2 j = 1 over ±1 n and deg f i + deg g i d and deg h i d/2 K d 1 derivation of unsatisfiable constraint 1 0 from f 1 = 0,, f m = 0 over ±1 n f f 1f2 f m D if 1 K d then separating hyperplane D with E D 1 = 1 and E D f 0 for all f K d K d = f = i f i g i + j h j 2

11 pseudo-distribution satisfies all local properties of ±1 n claim suppose f 0 is d/2-junta over ±1 n (depends on d/2 coordinates) then, E D f 0 proof: f has degree d/2 E D f = E D f 2 0 example: triangle inequalities over ±1 n E D x i x j 2 + xj x k 2 xi x k 2 0 corollary for any set S of d coordinates, marginal D = x S D is actual distribution D x S = x n S D x S, x n S = E D 1 xs 0 d-junta (also captured by LP methods, e.g., Sherali Adams hierarchies )

12 conditioning pseudo-distributions claim i n, σ ±1. D = x x j = σ D is deg.- d 2 pseudo-distr. proof D x = 1 P D x j =σ D x 1 x j =σ E D f 2 E D 1 xj =σ f2 = E D 1 xj =σ f 2 0 deg f (d 2)/2 deg 1 xj =σ f d/2 (also captured by LP methods, e.g., Sherali Adams hierarchies )

13 pseudo-covariances are covariances of distributions over R n claim there exists a (Gaussian) distr. ξ over R n such that E D x = E ξ and E D xx T = E ξξ T proof let μ = E D x and M = E D x μ x μ T consequence: E D q = E ξ q for every q of deg. 2 choose ξ to be Gaussian with mean μ and covariance M matrix M p.s.d. because v T Mv = E D v T x 2 0 for all v R n square of linear form

14 pseudo-distr. s satisfy (compositions of) low-deg. univariate properties claim for every univariate p 0 over R and every n-variate polynomial q with deg p deg q d, useful class of non-local E D p q x 0 higher-deg. inequalities enough to show: p is sum of squares p proof by induction on deg p choose: minimizer α of p p α 0 then: p= p α + x α 2 p for some polynomial P with deg p < deg p α R squares sum of squares by ind. hyp.

15 MAX CUT given: deg.-d pseudo-distr. D over ±1 n, satisfies L G = 1 ε goal: find y ±1 n with L G y 1 O ε [Goeman-Williamson] L G = 1 E G ij E G 1 x 2 i x j algorithm sample from Gaussian distr. ξ over R n with E ξξ T = E D xx T output y = sgn ξ analysis claim: P D x i x j = 1 η P y i y j 1 O η proof: ξ i, ξ j satisfies E ξ i ξ j = E D x i x j = 1 O η and Eξ i 2 = Eξ j 2 = 1 (tedious calculation) P sgn ξ i sgn ξ j 1 O η

16 low global correlation in (pseudo-)distributions [Barak-Raghavendra-S., Raghavendra-Tan] claim r. deg.- d 2r pseudo-distribution D, obtained by conditioning D, Avg i,j n I D x i, x j 1/r mutual information: I x, y = H x H x y proof potential Avg i n H x i ; greedily condition on variables to maximize potential decrease until global correlation is low how often do we need to condition? potential decrease Avg i n H x i Avg j n Avg i n H x i x j = Avg i,j n I D x i, x j only need to condition r times

17 MAX BISECTION d = 1/ε O 1 [Raghavendra-Tan] given: deg.-d pseudo-distr. D over ±1 n, satisfies L G = 1 ε, i x i = 0 goal: find y ±1 n with L G y 1 O ε and i y i = 0 algorithm let D be conditioning of D with global correlation ε O 1 sample Gaussian ξ with same deg.-2 moments as D output y with y i = sgn(ξ i t i ) (choose t i R so that E y i = E D x i ) analysis (t i = 0 is worst case same analysis as MAX CUT) almost as before: P D x i x j 1 η P y i y j 1 O η new: I x i, x j ε O 1 Ey i y j = E x i x j ± ε O(1) E i x i 2 = 0 E i y i E i y i 2 1/2 = E i x i 2 1/2 + ε O 1 n = ε O 1 n get bisection y from y by correcting ε O(1) fraction of vertices

18 SUM-OF-SQUARES method and approximation algorithms II David Steurer Cornell Cargese Workshop, 201

19 sparse vector given: linear subspace U R n (represented by some basis), parameter k n promise: v 0 U such that v 0 is k-sparse (and v 0 0, ±1 n ) goal: find k-sparse vector v U efficient approximation algorithm for k = Ω n would be major step toward refuting Khot s Unique Games Conjecture and improved guarantees for MAX CUT, VERTEX COVER, planted / average-case version (benchmark for unsupervised learning tasks) subspace U spanned by d 1 random vectors and some k-sparse vector v 0 previous best algorithms only work for very sparse vectors k n 1/ d [Spielman-Wang-Wright, Demanet-Hand] here: deg.- pseudo-distributions work for k n = Ω n up to d O n [Barak-Kelner-S.]

20 analytical proxy for sparsity (tight if v 0, ±1 n ) if vector v is k-sparse then v 1, v 2 v 1 k v 2 1, and v 1 k v 2 1 k limitations of l /l 1 (previous best algorithm; exact via linear programming) v = sum of d random ±1 vectors with same first coordinate v d, v 1 d + n d ratio l /l 1 algorithm fails for k n 1 d limitations of std. SDP relaxation for l 2 /l 1 (best proxy for sparsity) ideal object : distribution D over l 2 unit sphere of subspace U l 1 -constraint: E D v 2 1 k tractable relaxation: i,j E D v i v j k d n not a low-deg. polynomial in v unclear how to represent (also NP-hard in worst-case) [d'aspremont-el Ghaoui-Jordan-Lanckriet] but: for uniform distr. D over l 2 sphere of d-dim. rand. subspace i,j E D v i v j n d same limitation as l /l 1

21 degree-d SOS relaxation for l /l 2 deg.-d pseudo-distr. D: v U; v 2 = 1 R over unit l 2 -sphere of U notation: E D f v U; D f (only consider polynomials easy to integrate) v =1 non-negativity: E D h(v) 2 0 for every h of deg. d/2 normalization: E D 1 = 1 pseudo-distribution satisfies orthogonality: E D v 1 k v = 1/k g(v) = 0 for every g of deg. d

22 degree-d SOS relaxation for l /l 2 deg.-d pseudo-distr. D: v U; v 2 = 1 R over unit l 2 -sphere of U notation: E D f v U; D f (only consider polynomials easy to integrate) v =1 non-negativity: E D h(v) 2 0 for every h of deg. d/2 normalization: E D 1 = 1 pseudo-distribution satisfies orthogonality: E D v 1 k v = 1/k g(v) = 0 for every g of deg. d how to find pseudo-distributions? set of deg.-d pseudo-distributions = convex set with n O d -time separation oracle separation problem given: function D (represented as deg.-d polynomial) check: quadratic form f E D f 2 is p.s.d. or output violated constraint E D f 2 < 0

23 degree-d SOS relaxation for l /l 2 deg.-d pseudo-distr. D: v U; v 2 = 1 R over unit l 2 -sphere of U notation: E D f v U; D f (only consider polynomials easy to integrate) v =1 non-negativity: E D h(v) 2 0 for every h of deg. d/2 normalization: E D 1 = 1 pseudo-distribution satisfies orthogonality: E D v 1 k v = 1/k g(v) = 0 for every g of deg. d how to use pseudo-distributions? rule of thumb: set of deg.-d pseudo-moments E D f deg f d difficult* to distinguish / separate from deg.-d moments of actual distr. of solutions also: values E D f deg f > d do not carry additional information no need to look at them (* unless you invest n Ω d time to distinguish)

24 degree-d SOS relaxation for l /l 2 deg.-d pseudo-distr. D: v U; v 2 = 1 R over unit l 2 -sphere of U notation: E D f v U; D f (only consider polynomials easy to integrate) v =1 non-negativity: E D h(v) 2 0 for every h of deg. d/2 normalization: E D 1 = 1 pseudo-distribution satisfies orthogonality: E D v 1 k v = 1/k g(v) = 0 for every g of deg. d dual view (SOS certificates) v 1 k g + j h j 2 = 1 over v U; v 2 = 1 for some g of deg. d and {h j } of deg. d/2 no deg.-d pseudo-distr. exists ( no solution exists) for approximation algorithms: need pseudo-distr. to extract approx. solution (hard to exploit non-existence of SOS certificate directly)

25 general properties of pseudo-distributions let D = u, v be a deg.- pseudo-distribution over R n R n following inequalities hold as expected (same as for distributions) Cauchy Schwarz inequality E D u, v E D u 2 1/2 E D v 2 1/2 Hölder s inequality E D i u 3 i v i E D u 3/ E D v 1/ l -triangle inequality E D u + v E D u 1/ + E D v 1/

26 claim let U R n be a random d-dim. subspace with d let P be the orthogonal projector into U n then w.h.p, P v = O 1 n proof sketch v 2 j h j v 2 over v R n for h j s of deg. (SOS certificate for classical inequality P v O 1 [Barak-Brandao-Harrow-Kelner-S.-Zhou] n v 2 ) basis change: let x = B T v where B s columns are orthonormal basis of U (so that P = BB T ) P v = 1 n 2 i b i, x with b 1,, b n close to i.i.d. standard Gaussian vectors (so that E b b, x 2 = x 2 ) enough to show: 1 n i=1 n x 2 2 and E b b, x = 3 b i, x = O 1 E b b, x j h j x 2 reduce to deg. 2: 1 n i=1 n b i 2, y 2 O 1 E b b 2, y 2 (y = x 2 ) use concentration inequalities for quadratic forms (aka matrices)

27 approximation algorithm for planted sparse vector given: some basis of subspace U = span U v 0 R n, where U R n random d-dim. subspace, and v 0 R n with v 0 U, v 0 = 1, and v k 0 2 = 1 (e.g., k-sparse) goal: find unit vector w with w, v O k/n 1/ algorithm compute deg.- pseudo-distr. D = {v} over unit ball of U satisfying v = 1 k sample Gaussian distr. w with E ww T = E D vv T and renormalize analysis claim: E D v, v O k/n 1/ ( Gaussian w almost 1-dim.)

28 approximation algorithm for planted sparse vector given: some basis of subspace U = span U v 0 R n, where U R n random d-dim. subspace, and v 0 R n with v 0 U, v 0 = 1, and v k 0 2 = 1 (e.g., k-sparse) goal: find unit vector w with w, v O k/n 1/ analysis claim: E D v, v O k/n 1/ ( Gaussian w almost 1-dim.) 1 = k 1/ E D v 1/ (D satisfies = E D v, v 0 v 0 + P v 1/ (same function) v = E D v, v 0 v E D P v 1 (l -triangle inequ.) 1 E k 1 D v, v O 1 n 1/ (SOS cert. for U ) 1 k ) E D v, v 0 1 O k/n 1/ E D v, v O k/n 1/ (because v, v 0 = 1 P v 2 2 v, v 2 0 )

29 general properties of pseudo-distributions let D = u, v be a deg.- pseudo-distribution over R n R n following inequalities hold as expected (same as for distributions) Cauchy Schwarz inequality E D u, v E D u 2 1/2 E D v 2 1/2 Hölder s inequality E D i u 3 i v i E D u 3/ E D v 1/ l -triangle inequality E D u + v E D u 1/ + E D v 1/

30 products of pseudo-distributions claim suppose D, D : Ω R is deg.-d pseudo-distr. over Ω then, D D : Ω Ω R is deg.-d pseudo-distr. over Ω Ω proof tensor products of positive semidefinite matrices are positive semidefinite

31 let D = u, v be a deg.-2 pseudo-distribution over R n R n Cauchy Schwarz inequality E D u, v E D u 2 2 1/2 E D v 2 2 1/2 proof E D u, v 2 = E D D u, v u, v (D D is product pseudo-distr.) = E D D ij u i v i u j v j 1 2 E D D ij u i 2 v j 2 + ij u j 2 v i 2 = 1 2 E D D u 2 2 v u 2 2 v 2 2 = E D u 2 2 E D v 2 2 (2ab = a 2 + b 2 a b 2 ) (D D is product pseudo-distr.)

32 let D = u, v be a deg.- pseudo-distribution over R n R n Hölder s inequality E D i u i 3 v i E D u 3/ E D v 1/ proof E D i u i 3 v i E D i u i 1 2 E D i u i 2 v i 2 1/2 (Cauchy-Schwarz) E D i u i 1 2 E D i u i E D i v i 1/ (Cauchy-Schwarz) we also used: {u, v} deg- pseudo-distr. u u, u v deg.-2 pseudo-distr. (every deg.-2 poly. in u u, u v is deg.- poly. in u, v )

33 let D = u, v be a deg.- pseudo-distribution over R n R n l -triangle inequality E D u + v 1/ E D u 1/ + E D v 1/ proof expand u + v in terms of i u i, i u i 3 v i, i u i 2 v i 2, i u i v i 3, i v i bound pseudo-expect. of mixed terms using Cauchy-Schwarz / Hölder check that total is equal to right-hand side

34 tensor decomposition for simplicity: orthonormal and m = n given: goal: tensor T i a i (in spectral norm) for nice a 1,, a m R n find set of vectors B ±a 1,, ±a m approach show uniqueness : i a i i b i ±a 1,, ±a m ±b 1,, ±b m show that uniqueness proof translates to SOS certificate any pseudo-distribution over decomposition is concentrated on unique decomposition ±a 1,, ±a m recover decomposition by reweighing pseudo-distribution by log n degree polynomial (approximation to δ function)