arxiv: v3 [math.oc] 25 Nov 2015

Transcription

1 arxiv: v3 [math.oc] 5 Nov 015 On the von Neumann and Frank-Wolfe Algorithms with Away Steps Javier Peña Daniel Rodríguez Negar Soheili October 14, 018 Abstract The von Neumann algorithm is a simple coordinate-descent algorithm to determine whether the origin belongs to a polytope generated by a finite set of points. When the origin is in the interior of the polytope, the algorithm generates a sequence of points in the polytope that converges linearly to zero. The algorithm s rate of convergence depends on the radius of the largest ball around the origin contained in the polytope. We show that under the weaker condition that the origin is in the polytope, possibly on its boundary, a variant of the von Neumann algorithm that includes away steps generates a sequence of points in the polytope that converges linearly to zero. The new algorithm s rate of convergence depends on a certain geometric parameter of the polytope that extends the above radius but is always positive. Our linear convergence result and geometric insights also extend to a variant of the Frank-Wolfe algorithm with away steps for minimizing a convex quadratic function over a polytope. Tepper School of Business, Carnegie Mellon University, USA, jfp@andrew.cmu.edu Department of Mathematical Sciences, Carnegie Mellon University, USA, drod@cmu.edu College of Business Administration, University of Illinois at Chicago, USA, nazad@uic.edu 1

2 1 Introduction Assume A = [ ] a 1 a n R m n with a i = 1, i = 1,...,n. The von Neumann algorithm, communicated by von Neumann to Dantzig in the late 1940s and discussed later by Dantzig in an unpublished manuscript [7], is a simple algorithm to solve the feasibility problem: Is 0 conv(a) = conv{a 1,...,a n }? More precisely, the algorithm aims to find an approximate solution to the problem Ax = 0, x n 1 = {x R n + : x 1 = 1}. (1) The algorithm starts from an arbitrary point x 0 n 1. At the k-th iteration the algorithm updates the current trial solution x k n 1 as follows. First, it finds the column a j of A that forms the widest angle with y k := Ax k. If this angle is acute, i.e., A T y k > 0, then the algorithm halts as the vector y k separates the origin from conv(a). Otherwise the algorithm chooses x k+1 n 1 so that y k+1 := Ax k+1 is the minimum-norm convex combination of Ax k and a j. Let e j n 1 denote the n-dimensional vector with j-th component equal to one and all other components equal to zero. To ease notation, we shall write for throughout the paper. Von Neumann Algorithm 1. pick x 0 n 1 ; put y 0 := Ax 0 ; k := 0.. for k = 0,1,,... if A T y k > 0 then HALT: 0 conv(a) j := argmin a i,y k ; i=1,...,n θ k := argmin θ [0,1] { y k +θ(a j y k ) }; x k+1 := (1 θ k )x k +θ k e j ; y k+1 := Ax k+1 ; end for The von Neumann algorithm can be seen as a kind of coordinatedescent method for finding a solution to (1): At each iteration the algorithm judiciously selects a coordinate j and increases the weight ofthe j-th componentofx k while decreasingallofthe othersvia alinesearch step. Like other currently popular coordinate-descent and firstorder methods for convex optimization, the main attractive features of the von Neumann algorithm are its simplicity and low computational cost per iteration. Another attractive feature is its convergence rate. Epelman and Freund [8] showed that the speed of convergence of the

3 von Neumann algorithm can be characterized in terms of the following condition measure of the matrix A: ρ(a) := z R m, z =1 min a i,z. () i=1,...,n The condition measure ρ(a) was introduced by Goffin [13] and later independently studied by Cheung and Cucker [4]. The latter set of authors showed that ρ(a) is also a certain distance to ill-posedness in the spirit introduced and developed by Renegar [1, ]. Observe that ρ(a) can also be written as ρ(a) = z R m, z =1 min A T z,v = v n 1 z R m, z =1 min z,av. (3) v n 1 Hence ρ(a) > 0 if and only if 0 conv(a) and ρ(a) < 0 if and only if 0 int(conv(a)). When ρ(a) > 0, this condition measure is closely related to the concept of margin in binary classification [5] and with the minimum enclosing ball problem in computational geometry [6]. The quantity ρ(a) also has the following geometric interpretation as discussed in [3, Proposition 6.8]. If ρ(a) > 0 then from (3) and Lagrangian duality we get ρ(a) = z R m, z 1 = min min z,av v n 1 z,av v n 1 z R m, z 1 = min{ y : y conv(a)} = dist(0, conv(a)). (4) On the other hand, if ρ(a) 0 then (3) yields ρ(a) = ρ(a) = min z,av z R m, z =1v n 1 = {r : y r y conv(a)} = dist(0, conv(a)). (5) In either case ρ(a) = dist(0, conv(a)). Furthermore, observe that under the assumption A = [ a 1 a n ] R m n with a i = 1, i = 1,...,n it follows that z,av 1 for all z R m, z = 1 and v n 1. In particular, from (3) it follows that ρ(a) 1. Epelman and Freund [8] showed the following properties of the von Neumann algorithm. When ρ(a) < 0 the algorithm generates iterates x k n 1, k = 1,,... such that Ax k ( 1 ρ(a) ) k Ax0. (6) On the other hand, the iterates x k n 1 also satisfy Ax k 1 k as long as the algorithm has not halted. In particular, if ρ(a) > 0 3

4 then by (4) the algorithm must halt with a certificate of infeasibility A T 1 y k > 0 for 0 conv(a) in at most ρ(a) iterations. The latter bound is identical to a classical convergence bound for the perceptron algorithm [, 0]. This is not a coincidence as there is a nice duality between the perceptron and the von Neumann algorithms [19, 3]. We show that a variant of the von Neumann algorithm with away steps has the following stronger convergence properties. When 0 conv(a), possibly on its boundary, the algorithm generates a sequence x k n 1, k = 1,,... satisfying Ax k ) k/ (1 w(a) Ax 0. (7) 16 The quantity w(a) is a kind of relative width of conv(a) that is at least as large as ρ(a). However, unlike ρ(a) the relative width w(a) is positive for any non-zero matrix A R m n provided 0 conv(a). When ρ(a) > 0, or equivalently 0 conv(a), the von Neumann algorithm with away steps finds a certificate of infeasibility A T y k > 0 for 8 0 conv(a) in at most ρ(a) iterations. Figure 1 illustrates the different behavior of the von Neumann algorithmand [ ] the variant with awaysteps described in Section for A = The figure depicts the path of iterates {y k : k = 0,1,...} generated by each algorithm starting from y 0 = [ 1 0]. The zig-zagging behavior in the first case occurs because after the third iteration the search direction is nearly perpendicular to the current iterate and as a consequence the algorithm makes slow progress. By contrast, in the second case the away steps provide alternative search directions that enable the algorithm to make faster progress. The von Neumann algorithm can be seen as a special case of the Frank-Wolfe (also known as conditional gradient) algorithm [9, 16]. The von Neumann algorithm is also nearly identical to an algorithm for minimizing a quadratic form over a convex set independently developed by Gilbert [1]. The name Gilbert s algorithm appears to be more popular in the computational geometry literature [11]. We show that a linear convergence result similar to (7) also holds for a version of the Frank-Wolfe algorithm with away steps for minimizing a strongly convex quadratic function over a polytope. This variant of the Frank-Wolfe algorithm with away steps was introduced by Wolfe [6] and has been subsequently studied by various authors. In particular, linear convergence results similar to ours have been previously established in [10, 15, 16, 17] and more recently in [1]. Linear convergence results in the same spirit also hold for the randomized Kaczmarz algorithm[4] and for the methods of randomized coordinate 4

5 Figure 1: Iterates generated by the von Neumann algorithm (top) and its variant with away steps (bottom) descent and iterated projections [18]. The computational article [14] also reports numerical experiments for variants of the von Neumann algorithm with away steps. Our main contributions are the succinct and transparent proofs of linear convergence results that highlight the role of the relative width w(a) and a closely related restricted width φ(a). Our presentation unveils a deep connection between problem conditioning as encompassed by the quantities w(a), φ(a) and the behavior of the von Neumann and Frank-Wolfe algorithms with away steps. We also provide some lower bounds on w(a) and φ(a) in terms of certain radii quantities that naturally extend ρ(a). We note that the linear convergence results in [17] are stated in terms of a certain pyramidal width whose geometric intuition and properties appear to be less understood than those of w(a) and φ(a). The rest of the paper is organized as follows. In Section we describe a von Neumann Algorithm with Away Steps and establish its main convergence result in terms of the relative width w(a). Section 3 extends our main result to the more general problem of minimizing a quadratic function over the polytope conv(a). Finally, Section 4 discusses some properties of the relative and restricted widths. 5

6 Von Neumann Algorithm with Away Steps Throughout this section we assume A = [ ] a 1 a n R m n with a i = 1, i = 1,...,n. We next consider a variant of the von Neumann Algorithm that includes so-called away steps. To that end, at each iteration, in addition to a regular step the algorithm considers an alternative away step. Each of these away steps identifies a coordinate l such that the l-th component of x k is positive and decreases the weight of the l-th component of x k. The algorithm needs to keep track of the support, that is, the set of positive entries of a vector. To that end, given x R n +, let the support of x be defined as S(x) := {i {1,...,n} : x i > 0}. Von Neumann Algorithm with Away Steps 1. pick x 0 n 1 ; put y 0 := Ax 0 ; k := 0;.. for k = 0,1,,... if A T y k > 0 then HALT: 0 conv(a) j := argmin a i,y k ; l := arg a i,y k ; i=1,...,n i S(x k ) if a j y k,y k < y k a l,y k then (regular step) a := a j y k ; u := e j x k ; θ := 1 else (away step) a := y k a l ; u := x k e l ; θ := (x k) l 1 (x k ) l endif θ k := argmin { y k +θa }; θ [0,θ ] x k+1 := x k +θ k u; y k+1 := Ax k+1 end for Note that the above von Neumann Algorithm with Away Steps can also be applied to any non-zero matrix A = [ a 1 a n ]. The assumption that the columns of A are normalized, i.e., a i = 1, i = 1,...,n, simplifies our notation and exposition. In Section 3 below we extend our discussion to the case when the columns of A are not necessarily normalized. Observe that the iterates x k,k = 0,1,..., generated by the above von Neumann Algorithm with Away Steps satisfy x k n 1. This fact follows by induction: By construction, x 0 n 1. At iteration k we have x k+1 = x k +θ k u where x k n 1 and the components of u add up to zeroas uis either e j x k ore l x k. The bound θ k [0,θ ] in turn guarantees that x k+1 0 and so x k+1 1 = x k 1 = 1. 6

7 Define the relative width w(a) of conv(a) as { } w(a) := min Ax,al a j : l S(x), j {1,...,n}. x 0,Ax 0 l,j Ax (8) The next proposition shows that w(a) ρ(a) when 0 conv(a). To that end, observe that w(a) can also be written as { } Ax,Au Av w(a) = min : u,v n 1,S(u) S(x). x 0,Ax 0 u,v Ax (9) Proposition 1 If A is such that 0 conv(a) then w(a) ρ(a). Proof: Since 0 conv(a), equation (3) yields In particular, ρ(a) = z R m, z =1 min z,av 0. v n 1 ρ(a) = Hence from (9) we get min z, Av z R m, z =1v n 1 min Ax, Av. x 0,Ax 0v n 1 Ax (10) w(a) min Ax, Av ρ(a). x 0,Ax 0v n 1 Ax The first inequality holds because we can choose u = x x 1 in (9). The second inequality follows from (10). Observe that under the assumption A = [ ] a 1 a n R m n with a i = 1, i = 1,...,n it follows that Au Av for all u,v n 1. In particular, from (9) it follows that w(a). In Section 4 below we discuss some additional properties of w(a). In particular,wewillformallyprovethatw(a) > 0foranynonzeromatrix A R m n such that 0 conv(a). We are now ready to state the main properties of the von Neumann algorithm with away steps. Theorem 1 Assume x 0 n 1 is one of the extreme points of n 1. (a) If 0 conv(a) then the iterates x k n 1,y k = Ax k, k = 1,,... generated by the von Neumann Algorithm with Away Steps satisfy y k ) k/ (1 w(a) y

8 (b) The iterates x k n 1,y k = Ax k, k = 1,,... generated by the von Neumann Algorithm with Away Steps also satisfy y k 8 k as long as the algorithm has not halted. In particular, if 0 conv(a) then the von Neumann Algorithm with Away Steps finds a certificate of infeasibility A T y k > 0 for 0 conv(a) in at most 8 ρ(a) iterations. The crux of the proof of Theorem 1 is the following elementary lemma. Lemma 1 Assume a,y R m satisfy a,y < 0. Then min θ 0 y+θa = y a,y a, and the minimum is attained at θ = a,y a. Proof of Theorem 1: (a) The algorithm generates y k+1 by solving a problem of the form y k+1 = min θ [0,θ ] y k +θa where a = a j y k or a = y k a l is chosen so that a,y k = min{ y k a l,y k, a j y k,y k }. In particular, a,y k 1 ( a l y k,y k + y k a j,y k ) = 1 a l a j,y k (11) 1 w(a) y k. If θ k < θ then Lemma 1 applied to y := y k yields y k+1 = y k a,y k a y k w(a) 16 y k. The second inequality follows from (11) and a 1+ y k. Thus each time the algorithm performs an iteration with θ k < θ, the value of y k decreasesat least by the factor 1 w(a) 16. Toconclude, itsufficestoshowthataftern iterationsthenumber ofiterationswithθ k < θ isatleastn/. Tothatend, weapply the following argument from [17]: Observe that when θ k = θ 8

9 wehave S(x k+1 ) < S(x k ). On the other hand, when θ k < θ wehave S(x k+1 ) S(x k ) +1. Since S(x 0 ) = 1 and S(x) 1 for every x n 1, after any number of iterations there must have been at least as many iterations with θ k < θ as there have been iterations with θ k = θ. Hence after N iterations, the number of iterations with θ k < θ is at least N/. (b) Proceed as above but note that if the algorithm does not halt at the k-th iteration then a,y k a j y k,y k y k. Thus each time the algorithm performs an iteration with θ k < θ, we have y k+1 y k a,y k a y k y k 4 4. (1) Assume the algorithm has not halted after N iterations. Let m be the number of iterations with θ k < θ up to iteration N. If y N 4 m and θ N < θ then from (1) we get y N+1 4 m 4 m = 4(m 1) m 4 m+1. It follows by induction that if the algorithm has not halted after N iterations then y N 4 m. As in part (a), it must be the case that m N and consequently y N 8 N. Finally, if 0 conv(a) then ρ(a) = min{ y : y conv(a)} > 0 and so the algorithm must halt with a certificate of infeasibility A T y k > 0 8 for 0 conv(a) after at most ρ(a) iterations. 3 Frank-Wolfe Algorithm with Away Steps Throughout this section assume A = [ a 1 a n ] R m n is a nonzero matrix, and f(y) = 1 y,qy + b,y for a symmetric positive definite matrix Q R m m and b R m. Consider the problem min f(y) min f(ax). (13) y conv(a) x n 1 Observe that in contrast to Section, we do not assume that the columns of A are normalized in this section. Problem (1) can be seen as a special case of (13) when Q = I and b = 0. The von Neumann Algorithm can also be seen as a special case of the Frank-Wolfe Algorithm [9] for (13). This section extends the ideas and results from Section to the following variant of the Frank-Wolfe algorithm with away steps. This variant can be traced 9

10 back to Wolfe [6]. It has been a subject of study in a number of papers [1, 10, 14, 15, 17]. Frank-Wolfe Algorithm with Away Steps 1. pick x 0 n 1 ; put y 0 := Ax 0 ; k := 0;.. for k = 0,1,,... j := argmin i=1,...,n a i, f(y k ) ; l := arg i S(x k ) a i, f(y k ) ; if a j y k, f(y k ) < y k a l, f(y k ) then (regular step) a := a j y k ; u := e j x k ; θ := 1 else (away step) a := y k a l ; u := x k e l ; θ := (x k) l 1 (x k ) l endif θ k := argmin f(y k +θa) θ [0,θ ] x k+1 := x k +θ k u; y k+1 := Ax k+1 end for Observe that the computation of θ k in the second to last step reduces to minimizing a one-dimensional convex quadratic function over the interval [0,θ ]. WenextpresentageneralversionofTheorem1fortheaboveFrank- Wolfe Algorithm with Away Steps. The linear convergence result depends on a certain restricted width and diameter defined as follows. For x 0 with Ax 0 let φ(a,x) := { sup λ > 0 : u,v n 1, S(u) S(x), Au Av = λ } Ax Ax. Define the restricted width φ(a) and diameter d(a) of conv(a) as follows. φ(a) := min{φ(a,x) : x 0, Ax 0}, (14) x and Observe that for x 0 with Ax 0 { Ax,Au Av φ(a,x) u,v Ax d(a) := x,u n 1 Ax Au. (15) } : u,v n 1,S(u) S(x). Thus (9) and (14) imply that w(a) φ(a) for all nonzero A R m n. Furthermore, the restricted width φ(a) can be seen as an extension 10

11 of the radius ρ(a) defined in (). Indeed, when 0 int(conv(a)), we have span(a) = R m. Hence (5) can alternatively be written as { ρ(a) := min λ : v n 1, Av = λ } x 0,Ax 0 Ax Ax. This implies that φ(a,x) ρ(a) + Ax x 1 for all x 0 with Ax 0. Hence the following inequality readily follows φ(a) ρ(a). Section 4 presents a stronger lower bound on φ(a) in terms of certain variants of ρ(a). In particular, we will show that φ(a) > 0, and consequently w(a) > 0, for any nonzero matrix A R m n such that 0 conv(a). The linear convergence property of the von Neumann algorithm with away steps, as stated in Theorem 1(a), extends as follows. Theorem Assume x n 1 is a minimizer of (13). Let y = Ax and Ā := Q1/[ a 1 y a n y ]. If x 0 n 1 is one of the extreme points of n 1 then the iterates x k n 1,y k = Ax k, k = 1,,... generated by the Frank-Wolfe Algorithm with Away Steps satisfy ) k/ f(y k ) f(y ) (1 φ(ā) (f(y 0 ) f(y )). (16) 4d(Ā) The proof of Theorem relies on the following two lemmas. The first one is similar to Lemma 1 and also follows via a straightforward calculation. Lemma Assume f is as above and a,y R m satisfy a, f(y) < 0. Then a, f(y) minf(y +θa) = f(y) θ 0 a,qa, and the minimum is attained at θ = a, f(y) a,qa. Lemma 3 Assume f,a,y,ā all x n 1 are as in Theorem above. Then for l S(x),j=1,...,n f(ax),a l a j φ(ā) (f(ax) f(y )). Proof: Let y := Ax conv(a). Assume y y as otherwise there is nothing to show. Since y minimizes (13), we have f(y ),y y 0. For ease of notation put δ := f(y ),y y and y y Q := y y,q(y y ). It readily follows that f(y),y y = y y Q +δ 0, 11

12 and Hence, (f(y) f(y )) = y y Q +δ. f(y),y y = ( y y Q +δ) y y Q ( y y Q +δ) = y y Q (f(y) f(y )). Thus f(y),y y y y Q (f(y) f(y )). (17) On the other hand, by the definition of φ(a) there exist u,v n 1 λ with S(u) S(x) and λ φ(ā) such that Āu Āv = Āx Āx. Since Āx = Q 1/ (Ax y ) = Q 1/ (y y ), the latter equation can be rewritten as λ Au Av = y y (y y ). (18) Q Putting (17) and (18) together we get f(y),au Av = λ f(y),y y y y Q φ(ā) (f(y) f(y )). To finish, observe that f(ax),a l a j f(y),au Av l S(x),j=1,...,n φ(ā) (f(ax) f(y )). Proof of Theorem : This is a modification of the proof of Theorem 1(a). At iteration k the algorithm yields y k+1 such that f(y k+1 ) = min θ [0,θ ] f(y k +θa) where a = a j y k or a = y k a l, and f(y k ),a > 1 f(y k),a l a j 1 φ(ā) (f(y k ) f(y ). The second inequality above follows from Lemma 3. If θ k < θ then Lemma applied to y := y k yields f(y k+1 ) = f(y k ) a, f(y k) a,qa f(y k ) φ(ā) 4d(Ā)(f(y k) f(y )). 1

13 That is, ) f(y k+1 ) f(y ) (1 φ(ā) (f(y k ) f(y )). 4d(Ā) Then proceeding as in the last part of the proof of Theorem 1(a) we obtain (16). Remark 1 A closer look at the proof of Theorem reveals that the convergence bound (16) can be sharpened as follows: Replace φ(ā) with w f (A) φ(ā), where w f(a) is the following extension of w(a): w f (A) := min x n 1 Ax y l,j { } f(ax),a l a j (f(ax) f(y )) : l S(x),j {1,...,n}. In the special case when Q = I,b = 0 problem (13) specializes to problem (1). In this case if 0 conv(a) then we have y = 0 and w f (A) = w(a). Hence the sharpened version of Theorem yields y k ) k/ (1 w(a) 4d(A) y 0. If in addition the columns of A are normalized then d(a) and we recover the bound in Theorem 1(a). We have the following related conjecture concerning w(a) and φ(a). Conjecture 1 If A R m n is non-zero and 0 conv(a) then φ(a) = w(a). 4 Some properties of the restricted width Throughout this section assume A R m n is a nonzero matrix. As we noted in Section 3 above, w(a) φ(a) and φ(a) ρ(a) when 0 int(conv(a)). Our next result establishes a stronger lower bound on φ(a) in terms of some quantities that generalize ρ(a) to the case when 0 conv(a). To that end, we recall some terminology and results from [5]. Assume A = [ ] a 1 a n R m n is a non-zero matrix. Then there exists a unique partition B N = {1,...,n} such that both A B x B = 0, x B > 0 and A T N y > 0, AT By = 0 are feasible. In particular, B if and only if 0 conv(a). Also N if and only if 0 relint(conv(a)). Furthermore, if a i = 0 then i B. The above canonical partition (B,N) allows us to refine the quantity ρ(a) defined by () as follows. Let L := span(a B ) and L := {v 13

14 R m : v,y = 0 for all y L}. By convention, L = {0} and L = R m when B =. If L {0}, let ρ B (A) be defined as ρ B (A) := z L, z =1 min i B a i,z. Observe that if B, then L = {0} only when a i = 0 for all i B. If N, let ρ N (A) be defined as ρ N (A) := z L, z =1 min a i,z. i N When L {0}, it can be shown [5] that ρ B (A) < 0. Likewise, when N it can be shown that ρ N (A) > 0. In particular, the latter implies that ρ N (A) := min a i,z = min a i,z, (19) i N i N z L, z =1 z L, z 1 where a i is the orthogonal projection of a i onto L. Let A N denote the matrix obtained by projecting each of the columns of A N onto L. From (19) and Lagrangian duality it follows that ρ N (A) = min{ y : y conv(a N)}. (0) Similarly, it can be shown that if L {0} then ρ B (A) = {r : y L, y r y conv(a B )}. (1) Observe that (0) and (1) nicely extend (4) and (5). Indeed, (0) is identical to (4) when B =. Likewise, (1) is identical to (5) when N = and rank(a) = m. Furthermore, (0) and (1) imply that ρ N (A) = dist(0, conv(a N )) and ρ B(A) = dist L (0, conv(a B )) thereby extending the fact that ρ(a) = dist(0, conv(a)). The next results show that φ(a) can be bounded below in terms of ρ B (A) and ρ N (A). In particular, Corollary 1 shows that w(a) φ(a) > 0 whenever A 0 and 0 conv(a). Theorem 3 Assume A = [ a 1 a n ] R m n is a nonzero matrix. (a) If N = then L {0} and φ(a) ρ B (A). (b) If B = then φ(ā) ρ N(A) for Ā := [ A 0 ]. (c) If B and L = {0} then φ(a) ρ N (A). (d) If N and L {0} then φ(a) ρ B(A) ρ N (A) where A +ρ N (A), A = a i. i=1,...,n 14

15 Proof: (a) Assume x 0 is such that y := Ax 0. In this case y span(a B ) = L. Hence L {0} and by (1) there exists v n 1 and r ρ B (A) such that Av = r Ax Ax. Thus for u := x x 1 we have u,v n 1, S(u) S(x) and Au Av = ( ) r+ Ax 1 x 1 Ax Ax.Itfollowsthatφ(A,x) r+ Ax x 1 > ρ B (A). [ x (b) Assume x := 0 is such that y := Ā x = Ax 0. From (0) t] [ x ] it follows that Ax x 1 ρ N (A). Thus for u := x 1, v := e n+1 0 Ax 1 we have u,v n 1, S(u) S( x) and Āu Āv = x 1 Ax Ax. Ax It follows that φ(ā, x) x 1 ρ N (A). (c) Since B and L = {0}, it follows that A B = 0 and the columns of A N are precisely the non-zero columns of A. Thus from part (b) we get φ ([ A N 0 ]) ρ N (A). To finish, observe that φ(a) = φ( [ A N 0 ] ) because A B = 0. (d) Assume x 0 is such that y := Ax 0. Let L := span(a B ) and decompose y = y L + y where y = A N x N L and y L = A B x B + (A N A N )x N L. Put r := y y [0,1]. Assume r > 0 as otherwise y = y L span(a B ) and the statement holds with the better bound φ(a) ρ B (A) by proceeding exactly as in part (a). Since r > 0, we have x N 0. Put r N := y x N 1. From (0) it follows that r N ρ N (A). Next, ( 1 put w := x (AN N 1 A N )x ) N y L. Observe that w a i a i + y L A + r N 1 r and w L. Hence i N x N 1 r by (1) there exists x B 0, x B 1 = 1 such that A B x B = cw, where c := Taking x N := c x N 1 x N we get A N x N A B x B = ρ B (A) r r A +r N 1 r (0,1). c x N 1 (y +y L ) = ρ B (A) r N r A +r N 1 r y y. Thus letting u := (1 c)x + (0, x N ), v = ( x B,0) we get u,v n 1, S(u) S(x) and ( ) ρ B (A) r N Ax Au Av = (1 c) Ax + r A +r N 1 r Ax. () 15

16 Next, observe that ρ B (A) r N (1 c) Ax + r A +r N 1 r ρ B (A) r N r A +r N 1 r ρ B (A) r N A +rn ρ B (A) ρ N (A) A +ρ N (A). (3) The first inequality above follows because c (0,1), the second one follows from r [0,1] ( r A +r N 1 r ) = A +r N, andthethirdonefollowsfromr N ρ N (A). Putting()and(3) together we get φ(a,x) ρ B(A) ρ N (A) A +ρ N (A). Corollary 1 Assume A = [ a 1 a n ] R m n is a nonzero matrix and 0 conv(a). Then w(a) φ(a) > 0. Proof: Apply Theorem 3. Since 0 conv(a), we have B and thus case(b) cannot occur. Ifcase(a) occursthen φ(a) ρ B (A) > 0since ρ B (A) < 0 as L {0}. If case (c) occurs then φ(a) ρ N (A) > 0. Finally, if case (d) occurs then φ(a) ρ B(A) ρ N (A) > 0, since A +ρ N (A) both ρ N (A) > 0 and ρ B (A) < 0 as L {0}. To finish, recall that w(a) φ(a) as established in Section 3. We conclude with a few small examples that illustrate the values of φ(a), ρ B (A),ρ N (A) and their connection with the bounds in Theorem 3 for the three possible cases: N =, B =, and both B,N. Example 1 Assume ǫ,δ (0,1) and let [ ] A =. ǫ ǫ ǫ ǫ ǫδ ǫδ In this case B = {1,,3,4,5,6}, N =. It is easy to see that ρ B (A) = ǫ and φ(a) = φ(a, x) = (1+δ)ǫ for x = [ / 1/ ] T. [ ] 1 1 Example Assume δ (0,1) and let A =. In this case δ δ B =, N = {1,}. It is easy to see that ρ N (A) = δ and if we put Ā = [ A 0 ] then φ(ā) = φ(ā, x) = δ for x = [ 1/ 1/ 0 ] T. 16

17 Example 3 Assume ǫ,δ (0,1) and let A = ǫ ǫ ǫ ǫ δ δ In this case B = {1,,3,4}, N = {5,6}. It is easy to see that [ ] T 1 1 ǫ 0 0 (1+ǫ) (1+ǫ) 0 1+ǫ ρ B (A) = ǫ, ρ N (A) = δ. For x = we get A x = 0 0 ǫδ 1+ǫ It thus follows that φ(a) φ(a, x) = ǫδ 1+ǫ. On the other hand, Theorem 3 implies that in this case φ(a) ǫδ (1+ǫ,1+δ )+δ. In particular, ǫδ < φ(a) < ǫδ. Acknowledgements We are grateful to Simon Lacoste-Julien and Martin Jaggi for their comments on a preliminary draft of this paper. We are also grateful to two anonymous referees for their numerous constructive suggestions. The first author s research has been supported by NSF grant CMMI References [1] A. Beck and S. Shtern. Linearly convergent away-step conditional gradient for non-strongly convex functions. Technical report, Faculty of Industrial Engineering and Management, Technion, [] H. D. Block. The perceptron: A model for brain functioning. Reviews of Modern Physics, 34:13 135, 196. [3] P. Bürgisser and F. Cucker. Condition. Springer Berlin Heidelberg, 013. [4] D. Cheung and F. Cucker. A new condition number for linear programming. Math. Prog., 91(): , 001. [5] D. Cheung, F. Cucker, and J. Peña. On strata of degenerate polyhedral cones I: Condition and distance to strata. Eur. J. Oper. Res., 19(198):3 8, 009. [6] K. Clarkson. Coresets, sparse greedy approximation, and the Frank-Wolfe algorithm. ACM Transactions on Algorithms (TALG), 6(4):63,

18 [7] G.B. Dantzig. An ǫ-precise feasible solution to a linear program with a convexity constraint in 1 ǫ iterations independent of problem size. Technical report, Stanford University, 199. [8] M. Epelman and R. M. Freund. Condition number complexity of an elementary algorithm for computing a reliable solution of a conic linear system. Math. Program., 88(3): , 000. [9] M. Frank and P. Wolfe. An algorithm for quadratic programming. Naval Research Quarterly, 3:95 110, [10] D. Garber and E. Hazan. A linearly convergent conditional gradient algorithm with applications to online and stochastic optimization. Technical report, Technical report, Faculty of Industrial Engineering and Management, Technion, 013. [11] B. Gärtner and M. Jaggi. Coresets for polytope distance. In SCG 09 Proceedings of the twenty-fifth annual symposium on Computational geometry, pages 33 4, 009. [1] E. Gilbert. An iterative procedure for computing the minimum of a quadratic form on a convex set. SIAM Journal on Control, 4(1):61 80, [13] J. Goffin. The relaxation method for solving systems of linear inequalities. Math. Oper. Res., 5: , [14] J. Gonçalves, R. Storer, and J. Gondzio. A family of linear programming algorithms based on an algorithm by von Neumann. Optimization Methods and Software, 4(3): , 009. [15] J. Guélat and P. Marcotte. Some comments on Wolfe s away step. Math. Program., 35: , [16] M. Jaggi. Revisiting Frank-Wolfe: Projection-free sparse convex optimization. In ICML, volume 8 of JMLR Proceedings, pages , 013. [17] S. Lacoste-Julien and M. Jaggi. On the global linear convergence of Frank-Wolfe optimization variants. In Advances in Neural Information Processing Systems (NIPS), 015. [18] D. Leventhal and A. Lewis. Randomized methods for linear constraints: Convergence rates and conditioning. Math. Oper. Res., 35: , 010. [19] D. Li and T. Terlaky. The duality between the perceptron algorithm and the von Neumann algorithm. In Modeling and Optimization: Theory and Applications (MOPTA) Conference, 013. [0] A. B. J. Novikoff. On convergence proofs on perceptrons. In Proceedings of the Symposium on the Mathematical Theory of Automata, volume XII, pages 615 6,

19 [1] J. Renegar. Incorporating condition measures into the complexity theory of linear programming. SIAM J. on Optim., 5:506 54, [] J. Renegar. Linear programming, complexity theory and elementary functional analysis. Math. Program., 70:79 351, [3] N. Soheili and J. Peña. A primal dual smooth perceptron von Neumann algorithm. In K. Bezdek, Y. Ye, and A. Deza, editors, Fields Institute Communications Series on Discrete Geometry and Optimization, volume 69, pages Springer, 013. [4] T. Strohmer and R. Vershynin. A randomized Kaczmarz algorithm with exponential convergence. J. Fourier Anal. Appl., 15:6 5, 009. [5] V. Vapnik. Statistical Learning Theory. Wiley, [6] P. Wolfe. Convergence theory in nonlinear programming. In Integer and Nonlinear Programming. North-Holland, Amsterdam,