CSE 599: Interplay between Convex Optmzaton and Geometry Wnter 2018 Lecturer: Yn Tat Lee Lecture 17: Lee-Sdford Barrer Dsclamer: Please tell me any mstake you notced. In ths lecture, we talk about the Lee-Sdford barrer (apparently, I have no talent n namng). My orgnal motve to study lnear programmng and convex optmzaton s to solve the maxmum flow problem faster. When we use the standard logarthmc barrer functon to solve the maxmum flow problem, we get a O ( m) teratons algorthm where m s the number of edges. Each teraton nvolves solvng a Laplacan system and hence takes nearly lnear tme. Therefore, we get a O (m 1.5 ) tme algorthm, matchng the prevous best algorthm by Goldberg and Rao [91]. Problem 17.0.2. What s the relaton between nteror pont methods and Goldberg and Rao algorthm? Last lecture, we showed that n general one can get an O ( n) teratons algorthm for lnear programs and hence naturally one conjectured that a O (m n) tme algorthm for the maxmum flow problem s possble where n s the number of vertces. To get the O (m n) tme algorthm, we need to overcome the followng hurdles: 1. Desgn a O (n)-self concordant barrer that can be computed very effcently. 2. Modfy the algorthm to work on the lnear program of the form mn c x. B f=d,0 f u Note that ths lnear program does not have O (n)-self concordant barrer. However, ths s not a proof that we cannot desgn an O ( n) teratons algorthm. In partcular, f there s no the upper constrant f u, the dual lnear program has n varables and hence we can run the nteror pont method on the dual. Wth the upper constrant, what we dd s smply move the dual algorthm to prmal and modfed t such that t works wth the upper constrant. For smplcty, we wll only focus on gettng a O (n)-self concordant barrer that can be computed pretty effcently. 17.1 Problem of Logarthmc barrer Before we talk about our barrer, let us understand the problem of the logarthm barrer. It s known that nteror pont methods on logarthmc barrer can take O( m) even f m s exponental to n. (See for example [92]). One reason s that we can perturb the barrer arbtrary by repeatng constrants. Suppose we repeat the th constrant w many tmes, essentally our barrer functon becomes w ln(a x b ). By choosng w approprately, one can make the central path nearly every vertces of cube lke the followng For these polytopes, the easest way to fx t s to remove redundant constrants. However, when there are many constrants wth smlar but not dentcal drecton, then t s less clear how to handle. 17-1
17-2 Lecture 17: Lee-Sdford Barrer 17.2 Volumetrc Barrer Fgure 17.1: Coped from [92]. Please gnore the labels. Vadya proposed the volumetrc barrer [94]. The self-concordance of ths barrer has a better dependence on m compared to the log-barrer. The volumetrc barrer s φ 2 (x) = lndet(a S 2 A) where s = a x b and S = dag(s). By smple calculatons, we have the followng: Lemma 17.2.1. We have that φ 2 (x) = A xσ x, 2 φ 2 (x) = A x(6σ x 4P (2) x ))A x. where A x = S 1 A, P x = A x(a xa x ) 1 A x, P x (2) s the Schur product of P, σ x s the dagonal of P x as a vector, and Σ x s the dagonal of P x as a dagonal matrx. Furthermore, we have that 2A x Σ xa x 2 φ 2 (x) 6A x Σ xa x. Intutvely, one can thnk φ 2 (x) σ x, ln(a x b ). Instead of a formal proof, we show how to estmate the self-concordance of φ 2 : Lemma 17.2.2. Consder φ(x) = w ln(a x b ). Then, φ(x) ( 2 φ(x)) 1 φ(x) w and that D 3 φ(x)[h,h,h] 2max σ ( WA x ) w ( D 2 φ(x)[h,h] ) 3/2. where σ ( WA x ) s the leverage score of A x W, namely σ ( WA x ) = ( WA x (A xwa x ) 1 Ax W). Remark. Geometrcally, σ( WA x) w = max h x 1 a h, s thedstancefromxtotheboundaryof th constrant. Proof. Note that φ(x) = A xw and 2 φ(x) = A xwa x. Hence, we have φ(x) ( 2 φ(x)) 1 φ(x) = w A x (A x WA x) 1 A x w w w = w
Lecture 17: Lee-Sdford Barrer 17-3 where we used that WA x (A x WA x) 1 A x W s an orthogonal projecton matrx. For the second condton, we have D 3 φ(x)[h,h,h] = 2 w ( a h ) 3 2 w ( a h ) 2 max a h s s s. Note that Hence, we have a h s = a (A xwa x ) 1 2(A xwa x ) 2h 1 s ( A x (A xwa x ) 1 A ) x h x. D 3 φ(x)[h,h,h] 2max σ ( WA x ) w h 3 x By rescalng the φ by constant, we obtan a barrer wth the self-concordance at x σ ( WA x ) w max w. Now, we analyze the case w = σ (A x ). Lemma 17.2.3. We have that σ (A x ) max σ ( Σ (A x )A x ) σ (A x ) 2n m. Proof. By the property of leverage score, we have that σ (A x ) n. We bound the maxmum term n two ways. Frst, we note that Second, we have that whch mples that Therefore, we have Hence, we have σ ( Σ (A x )A x ) σ (A x ) 1 σ (A x ). tr((a x A x) 1 A x dag(1 σ 1 2m )A x) σ 1 2m A x dag(1 σ 1 2m )A x 1 2 A x A x. σ 1 2 A xa x 2A xdag(1 σ 1 2m )A x 4m A xσa x. σ ( Σ A x ) ( ΣA x (A xσa x ) 1 Ax Σ) 4m( ΣA x (A xa x ) 1 Ax Σ) = 4mσ (A x ) 2. Combnng both cases, we have σ ( Σ (A x )A x ) σ (A x ) 1 mn( σ (A x ),4mσ (A x )) 2 m.
17-4 Lecture 17: Lee-Sdford Barrer Formally, the weghng depends on x and hence one need to compute the dervatves formally. By some calculaton, we ndeed can prove that φ 2 has self concordance O(n m). Note that ths barrer has less dependence on m. In the same paper, Vadya consder the barrer φ 2 n m lns and proved that t has self-concordance O( mn). You can agan estmate the self-concordance of ths hybrd barrer by the calculaton above. 17.3 Lee-Sdford Barrer Intutvely, we can thnk the volumetrc barrer rewegh the constrants by leverage score. Lemma 17.2.2 suggests that the best barrer should have σ( WA x) w all roughly the same. Ideally, one would set w = σ ( WA x ). Ths s exactly the condton of the l Lews weght. Recall from Defnton 11.1.3 that l p Lews weght w p (A) be the unque vector w R m + such that ) w = σ (W 1 2 1 p A. Now, the problem s smply to fnd a barrer functon φ such that at every x, we have φ p (x) w p (A x )ln(a x b ). Note that f we ndeed has such functon for p =, then ths barrer should have self-concordant O(n). Recall from Lemma 11.1.5 that w p (A x ) s the maxmzer of ( ) φ p (x) = max logdet A 2 x w W1 p Ax 0 (1 2 m q ) w. Note that ths functon looks very close to volumetrc barrer, except that t optmzes over a famly of volumetrc barrer wth dfferent reweghng. Therefore, t s natural to ask f φ p (x) s self-concordant. Note that φ 2 (x) s exactly the volumetrc barrer. Also, one can prove that w 0 (A x ) s constant (f A s n general poston) and hence φ 0 (x) looks lke the log barrer functon. Therefore, φ p can thnk as a famly of barrer that ncludes log barrer and volumetrc barrer. By some calculatons, t turns out φ p s self-concordance for all 0 < p <. Theorem 17.3.1. Gven a polytope Ω = {Ax > b}. Suppose that Ω s bounded and non-empty and that every row of A s non-zero. For any p > 0, after rescalng, the φ p (x) s a O p (n m 1 p+2 ) self-concordant barrer. Take p = Θ(logm), we have a barrer wth self concordance s O(nlog O(1) m). Unfortunately, the proof of ths s a lttle bt long. We left t as a 12-pages long calculus exercse. =1 17.4 Open Problems Ideally, one would ask f the LS barrer s practcal. Snce the current best approach n practce s the prmal dual central path, we have the followng queston: Problem 17.4.1. What s the prmal dual analogy of the LS result?
Lecture 17: Lee-Sdford Barrer 17-5 Besde lnear programs, another mportant class of problems s sem-defnte programmng. Nesterov and Nemrovsk generalzed the volumetrc barrer to semdefnte programmng settng [93]. Naturally, can we generalze the LS barrer to that settng? Note that the current fastest algorthm for sem-defnte programmng s cuttng plane method nstead of nteror pont method. Ths s very unusual for structured convex programmng. Therefore, there are a lot of opportuntes for mprovng algorthms on sem-defnte programmng. Problem 17.4.2. Improve the convergence rate of nteror pont methods for sem-defnte programmng. If we vew the LS barrer as a weghted combnaton of log barrer, then t s natural to ask: Problem 17.4.3. Gven barrers φ for K, can we come wth a barrer for K that s better than φ? Fnally, gong back to my orgnal purpose of solvng lnear programs: Problem 17.4.4. Can you solve the maxmum flow problem faster than O (m n)? References [91] Andrew V Goldberg and Satsh Rao. Beyond the flow decomposton barrer. Journal of the ACM (JACM), 45(5):783 797, 1998. [92] Murat Mut and Tamás Terlaky. A tght teraton-complexty upper bound for the mty predctorcorrector algorthm va redundant klee-mnty cubes. 2014. [93] Yur Nesterov and Arkad Nemrovsk. Interor-pont polynomal algorthms n convex programmng. SIAM, 1994. [94] Pravn M Vadya. A new algorthm for mnmzng convex functons over convex sets. In Foundatons of Computer Scence, 1989., 30th Annual Symposum on, pages 338 343. IEEE, 1989.