15-850: Advanced Algorithms CMU, Fall 2018 HW #4 (out October 17, 2018) Due: October 28, 2018

Transcription

1 15-850: Advanced Algorithms CMU, Fall 2018 HW #4 (out October 17, 2018) Due: October 28, 2018 Usual rules. :) Exercises 1. Lots of Flows. Suppose you wanted to find an approximate solution to the following multicommodity flow problem: given a digraph G = (V, E) with unit arc capacities, send F i flow from node s i to node t i in the graph, for all i [k]. You should imagine that the flow from s i to t i is of commodity i (e.g., oil, water, sand...) which are all distinct. (a) Suppose P i is the set of all paths from s i to t i : show that the following LP captures the problem we are trying to solve. The variables are f P, one for each path in i P i. i P P i f P = F i P P i :e P f P 1 f 0 i [k] e E (b) Given weights q m, how would you solve the average LP for this problem? Show you can find a flow that satisfies the demands, but uses at most (1 + ε) capacity on each edge, in time ( i F i) = O(k(m+n log n)) poly(ε) O(km(m+n log n)) poly(ε). 2. Zero-Sum Games using LP Duality. Recall the zero-sum game setup: we re given a matrix M R m n ; if the row player plays a strategy x m and the column player plays strategy y n, the payoff to the row player is x My. If we define C(x) = min y n x My, and R(y) = max x m x My, the von Neumann minimax theorem proves that (a) for all x, y, C(x) R(y), and moreover (b) there exist x, y such that C(x ) = R(y ). (a) Show an LP to compute max x C(x), the optimal strategy for the row player. (Hint: be careful, the definition of C() has a min sitting in there, so you re looking to find max x min y x My, which certainly does not look like a linear program.) (b) Show an LP to compute min y R(y), the optimal strategy for the column player. (c) Show that you can, in fact, find LPs for both the above parts, such that the dual of the first LP is a solution to the second part. (d) Use weak duality to infer the first part of the minimax theorem, and strong duality to infer the second part. 3. Strength in Convexity. A function f : R n R is called α-strongly-convex with respect to norm if f(y) f(x) + f(x), y x + α 2 y x 2. I.e., if the function is not just convex, but locally it grows at least as fast as a quadratic. 1

2 For the rest of this proof, focus on just the Euclidean norm 2, and assume that f(x) G. Modify the basic gradient descent analysis to show that the update rule x t+1 x t η t f(x t ) with suitably chosen η t can be used to find x R n with ( G f( x) f(x 2 ) log T ) O. α T Observe: the assumption of strong convexity gives better convergence guarantees (i.e., the dependence on T is better, and there is no dependence on D = x 0 x. Show that this analysis also works in the online case to give a regret bound, if each function is strongly convex. (A slightly harder problem is to remove the log T term in the numerator in the offline case. Why does this improvement not extend to the online case?) 4. These are Small Numbers. For an integer k, define k = 1 + log 2 ( k + 1) ; for a rational p/q (with p, q coprime, q > 0), define p/q = p + q ; for a matrix R = (r ij ) of rationals, define M = i,j r ij. Let det(r) denote the determinant of R. (a) If R is an n n matrix, show that det(r) poly(n, R ). Now consider the LP min{c T x Ax b}, where x R n, and A R m n with m n. (b) If each of the numbers in A and b are rationals having size at most S, and if x is a basic feasible solution, give an upper bound on the size of each entry of x. In particular, show that each entry x i is a rational number with size at most K = poly(n S). (c) If the LP also has a finite optimum, and the numbers in c have size at most S, infer that the optimal value of the LP has size O((K + S)n). Hint: Google for Cramer s rule and Hadamard Inequality. 5. That s the Norm. In lecture, we defined a differentiable convex function f : K R to be G-Lipschitz with respect to norm if f(x) f(y) x y G. Show that this is equivalent to f(x) G for all x K. Similarly, show that f being α-strongly-convex with respect to is equivalent to f(x) f(y) α x y. And that f being β-smooth with respect to is equivalent to f(x) f(y) β x y. 6. The Oracle Separates. Given a graph with edge costs c e, the following LP is called the Held-Karp/subtour elimination LP, and is a relaxation for the traveling salesman problem. Give a separation oracle for it. min e c ex e e S x e 2 0 x e 1. S V 2

3 Here s another LP with variables {x i } n i=1, non-negative costs {c i} n i=1, and sizes {s ij} n i,j=1 : again give a separation oracle. min i c ix i i S x is i, S S 0 x i 1. S [n] Problems Solve problem #2, and any two of the other three. 1. Capacitated Max-Flow and Width Reduction. Consider the directed s-t max-flow problem for general capacities (as opposed to unit capacities in lec #16-17). The LP is: P f P = F P :e P f P /c e 1 f P 0. e E For brevity, define f e := P :e P f P, so the constraints are f e /c e 1 for all e. Now given the weights p m from Hedge, the average constraint looks like e p e (f e /c e ) = e f e(p e /c e ) 1. Define K = {f P f P = F, f P 0 P } as the easy constraints, so MW tries to solve the average constraint w.r.t. solutions in K. (a) (Do not submit.) Suppose the oracle sends the entire F units of flow along a shortest path w.r.t. p e /c e. Show that there are capacitated networks, and possible p m, where all F flow is routed along a path using the least-capacity edge. Hence the width of this oracle is Ω(F/c min ). Recall that the width is the maximum violation of any constraint. Note that F may be as large as Ω(mc max ), so with general capacities, this ratio could be m. It turns out that adding a little bit to the edge weights (e.g., setting w e := p e + ε/m) reduces the width at the expense of giving slightly approximate solutions! For the rest, assume ε 1/10, say. Assume the instance is feasible; i.e., there exists a solution f to the LP. (b) Set the edge weights to be w e := p e +ε/m, compute the shortest path w.r.t. edge lengths w e c e, and send all F flow along it. If this shortest path is P, show (c) Show that pe f F e P c e min e f K e E c e w e 1 + ε. max F/c e P e O( m ε ). Hence the width of the average problem is O(m/ε). (d) (Do not submit.) Using this oracle and the MW algorithm guarantee, give an Õ(m2 /ε 3 )- time (1 + ε)-approximate max-flow algorithm for the capacitated case. 3

4 (e) (Do not submit.) use these ideas to get an algorithm for the multi-commodity case from exercise #1 that works for capacitated graphs, but whose runtime does not depend on the magnitude of the capacities. Note: This idea was used in the Christiano et al. paper, combined with the electrical flows, to bring the width down to O( m/ε). 2. Solving a Linear System. Given a positive-definite matrix A R n n with eigenvalues 0 < λ 1 λ 2... λ n, the condition number for A is κ := λn λ 1. Given a vector b R n, the goal is to find a near -solution to the linear system Ax = b. Consider the function f(x) = 1 2 x Ax bx. (a) (Do not submit) Show that f is a convex function, with f(x) = Ax b. Hence infer that the minimizer x of f(x) satisfies Ax = b. Moreover, show that f is λ 1 -strongly-convex and λ n -smooth. (b) Show that gradient descent on f( ) starting at some point x 0 R n guarantees that x t x 2 max{ µ 1, µ n } t x 0 x 2 where µ 1 µ n are the eigenvalues of (I ηa). (c) Show that a suitable choice of η ensures that x t x 2 δ x 0 x 2 after O(κ log δ 1 ) iterations. Use this to show that the error is after O(κ log λmax x 0 x 2 ε ) iterations. Ax t b 2 ε 3. Gradient Descent, meet Linear Optimization. In constrained gradient descent (over convex body K R n ), we took a step along the negative gradient, and then projected back to K. Another approach is to use linear optimization: given x t K, the next iterate is: y t arg min{( f(x t )) y y K} x t+1 (1 η t )x t + η t y t. The minimizer in the first step can be found using linear optimization over K, which may be much simpler than general convex optimization. Let s analyze this for the case where f is β-smooth with respect to norm i.e., f(y) f(x) + f(x), y x + β 2 y x 2. Your arguments for parts (a)-(c) should work for any norm. (a) Let D := max p,q K p q be the diameter of K. Use smoothness to show that for t 0: f(x t+1 ) f(x t ) η t f(x t ), x x t + β 2 η2 t D 2. (b) Define Φ t := t(f(x t ) f(x )). Use convexity and part (a) to show that Φ t+1 Φ t β 2 η2 t D 2 β 2 η td 2 if we set η t = 1 t+1. Hence, infer that after T O( βd2 2ε log 1 ε ). f(x T ) f(x ) + ε 4

5 (c) (Do not submit.) Show that if K is a polytope with m n non-trivial constraints (and n non-negativity constraints), then any vertex of this polytope has at most m non-zero coordinates. So if we start with x 0 = 0, the final solution x t is very sparse it has at most mt non-zero coordinates. E.g., for K = n := {x R n i x i = 1, x i 0}, x t has at most t non-zeroes. Now let us use this in an application. Given a topic matrix A R m n where each column is a topic and each row is a word, and given a document y R m, we want to find a small set of topics may generate this document. Formally, we want to find x R n s.t. y is close to Ax and x is sparse. Since x 0 minimization is hard, we use x 1 -minimization instead: i.e., we solve min x R y Ax 2 n 2 s.t. x 1 = 1. In this setting n m (there may be n = billions of topics, but only m 10 5 words), and we don t want to take poly(n) time! Suppose we have an oracle that given w R m can output arg max j [n] w, A j in Z m time here A j denotes the j th column of A. (d) If we use the above linear optimization-based algorithm, show that each iterate x t can be computed in O(Zt + t 2 ), by keeping track of just the non-zero entries in x t. (e) Show that if each column length A j 2 L then f(x) = y Ax 2 2 is 2L2 -smooth w.r.t. the l 1 -norm. (You may use Exercise 5 without proof.) Observe that the l 1 -diameter D of the polytope K = {x x 1 = 1} is 1. Hence we can get an ε-approximate solution in time O((ZL 2 /ε + L 4 /ε 2 ) log 2 1 ε ). 4. Hmm, That s Odd... To solve the max-weight perfect matching problem on general graphs, we need to optimize over the perfect matching polytope. In turn, this requires that we find a separation oracle for the odd-set constraints. I.e., given x R E, we wish to find a set S V such that S is odd, and x( S) is minimized, where x( S) := i S,j S x ij. Then, comparing this min-odd-cut value to 1, we can find a violated constraint (if one exists). Assume that V is even, else the LP has no feasible solution anyways. (a) A function f : 2 V R is called submodular if for all A, B V, we have f(a) + f(b) f(a B) + f(a B). Show f(a) := x( A) is submodular. Observe f is symmetric, i.e., f(a) = f(v \ A). (b) If (C, C) is the min-cut, i.e, if x( C) is the least among all non-empty cuts, and C is even, then show that there exists a min-odd cut contained within C, or within C. (c) Give an algorithm to find a min-odd-cut in polynomial time. 5