A Simple Sampling Lemma: Analysis and Applications in Geometric Optimization

Transcription

1 A Simple Sampling Lemma: Analysis and Applications in Geometic Optimization Bend Gätne and Emo Welzl Apil 2, 2001 Abstact andom sampling is an efficient method to deal with constained optimization poblems in computational geomety In a fist step, one finds the optimal solution subject to a andom subset of the constaints; in many cases, the expected numbe of constaints still violated by that solution is then significantly smalle than the oveall numbe of constaints that emain This phenomenon can be exploited in seveal ways, and typically esults in simple and asymptotically fast algoithms Vey often the analysis of andom sampling in this context boils down to a simple identity the sampling lemma which holds in a geneal famewok, yet has not been stated explicitly in the liteatue In the moe esticted but still geneal setting of LP-type poblems, we pove tail estimates fo the sampling lemma, giving Chenoff-type bounds fo the numbe of constaints violated by the solution of a andom subset As an application, we povide the fist theoetical analysis of multiple picing, a heuistic used in the simplex method fo linea pogamming in ode to educe a lage poblem to few small ones This follows fom ou analysis of a eduction scheme fo geneal LP-type poblems, which can be consideed as a simplification of an algoithm due to Clakson The simplified vesion needs less andom esouces and allows a Chenoff-type tail estimate 1 Intoduction andom sampling and andomized incemental constuction have become well-established, by now even classical, design paadigms in the field of computational geomety, cf [27] Many algoithms following that paadigm have been simplified to a point whee they can easily be taught in intoductoy CS couses, with almost no technical difficulties This was not always the case; pioneeing papes, notably the ones by Clakson and Sho [6, 9], Mulmuley [26], and by Guibas, Knuth and Shai [18], still equied moe technical deivations This changed when Seidel populaized the backwads analysis paadigm fo andomized algoithms [30] Togethe with the abstact famewok of configuation spaces, this technique allows to teat many diffeent algoithms in a simple and unified way [11] The goal of this pape is to populaize and pove esults aound a simple identity the sampling lemma which undelies the analysis of andomized algoithms fo many geometic The fist autho acknowledges suppot fom the Swiss Science Foundation SNF, poject No A peliminay vesion of this pape appeaed in the Poceedings of the 16th Annual ACM Symposium on Computational Geomety SCG, 2000, pp Institut fü Theoetische Infomatik, ETH Züich, ETH Zentum, CH-8092 Züich, Switzeland, gaetne@infethzch, emo@infethzch 1

2 optimization poblems By that we mean poblems defined in a low-dimensional space, which usually implies that they have few constaints o few vaiables when witten as mathematical pogams As we show below, special cases of the identity, o inequalities implied by it, ae used in many places, including the analysis of the geneal configuation space famewok To the knowledge of the authos, the identity itself, howeve, has not been noticed explicitly The Sampling Lemma Let S be a set of size n and φ a function that maps any set S to some value φ 1 Define V := {s S \ φ {s} φ}, X := {s φ \ {s} φ} V is the set of violatos of, while X is the set of exteme elements in Obviously, s violates s is exteme in {s} 1 Fo a andom sample of size, ie a set chosen unifomly at andom fom the set S of all -element subsets of S, we define andom vaiables V : V and X : X, and we conside the expected values v := EV, x := EX Lemma 11 Sampling Lemma Fo 0 < n, v n = x Poof: Using the definitions of v and x +1 as well as 1, we can ague as follows n v = [s violates ] S s S\ = = = S s S\ Q +1 S s Q n + 1 [s is exteme in {s}] [s is exteme in Q] x +1 Hee, [ ] is the indicato vaiable fo the event in backets Finally, n +1 / n = n / + 1 To appeciate the simplicity if not tiviality of the lemma, one should conside it as a special case of the following obsevation: given a bipatite gaph, the aveage vetex degee in clea 1 Hee, the only pupose of φ is to patition 2 S into equivalence classes; late, the function-notation becomes 2

3 one colo class times the size of that class equals the aveage vetex degee in the othe colo class times its size In ou case, the two colo classes ae the subsets of S of sizes and + 1, espectively, and two sets and {s} shae an edge if and only if s violates equivalently, if s is exteme in {s} This means, the sampling lemma still holds if violation is individually defined fo evey pai, s A situation of quite simila flavo, whee a simple bipatite gaph undelies a pobabilistic scenaio, has been studied by Dubhashi and anjan [12] We can also establish a vesion of the sampling lemma in the model of Benoulli sampling, whee is chosen by picking each element of S independently with some fixed pobability p [0, 1] we say, is a andom p-sample Let V p and X p denote the andom vaiables fo the numbe of violatos and exteme elements, espectively, in a p-sample, and let v p and x p be the coesponding expectations Lemma 12 p-sampling Lemma Fo 0 p 1, pv p = 1 px p Poof: Each -element set occus as a p-sample with pobability n p 1 p n Using the Sampling Lemma 11 it follows that n n 1 n pv p = p p 1 p n v = p =0 =0 n 1 n = p p 1 p n x +1 = p + 1 =0 =1 n n = 1 p p 1 p n x = 1 p =1 = 1 px p n p 1 p n v n n p 1 1 p n +1 x n =0 n p 1 p n x In the next section we discuss some well-known esults obtained by andom sampling and show that all of them easily follow fom the sampling espectively p-sampling lemma Concentating on the Sampling Lemma 11, we elaboate on its connection to configuation spaces and backwads analysis Section 3 deals with LP-type poblems, which can be consideed as functions φ with specific popeties Section 4 establishes Chenoff-type tail estimates fo the andom vaiable V, ie fo the numbe of violatos of a andom sample The sampling lemma and the tail estimates ae finally used in Section 5 to analyze an algoithm fo geneal LP-type poblem, which can be consideed as the pactical vesion of Clakson s eduction scheme [16] Its specialization to linea pogamming is a vaiant of multiple picing [5] 2 Incanations of the Sampling Lemmata Seaching in a soted compact list A soted compact list epesents a set S of n odeed keys in an aay, whee the ode among the keys is established by additional pointes linking each element to its pedecesso in the 3

4 ode, see Figue 1 It is well-known that the smallest key in a soted compact list can be found in O n expected time [10, Poblem 11-3] Figue 1: A soted list of 8 keys, compactly stoed in an aay Fo this, one daws a andom sample of = Θ n keys, finds the smallest key s 0 in the sample, and finally follows the links fom s 0 to the oveall smallest key The efficiency comes fom the fact that an expected numbe of only Θ n keys is still smalle than s 0 In geneal, setting φ = min and obseving that X +1 1, the sampling lemma yields E#{s S \ s < min} = n Note that s < min is equivalent to min {s} min Popety 2 was exploited by Seidel in the following obsevation: given a simple d-polytope P with n vetices, specified by its 1-skeleton the gaph of vetices and edges of P, one can find the vetex that minimizes some linea function f in expected time Od n The coesponding andomized suboutine seves as a building block of a simple algoithm fo computing the intesection of halfspaces, o dually, the convex hull of points in d-dimensional space Fo d 4, this algoithm achieves optimal expected wost-case pefomance [31] Smallest enclosing ball Conside the poblem of computing the smallest enclosing ball of a set S of n points in d- dimensional space, fo some fixed d andomized incemental algoithms do this in expected On time [33], based on the following fact: if the points ae added in andom ode, the pobability that the n-th point is outside the smallest enclosing ball of the fist n 1 points is bounded by d + 1/n In geneal, it holds that if S is a andom sample of points, and ball denotes the smallest enclosing ball of, then E#{p S \ p ball} d + 1 n Again, this follows fom the sampling lemma, with φ = ball, togethe with the obsevation that any set has at most d + 1 exteme elements [33], and the fact that s ball ball {s} ball Simila esults hold fo the smallest enclosing ellipsoid poblem The andomized incemental algoithm based on them was the fist one to achieve an expected untime of On fo that poblem, cf [33] The pioneeing applications of andomized incemental constuction along these lines wee Clakson s and Seidel s linea-time algoithms fo linea pogamming with a fixed numbe d of vaiables [8, 29] Plana convex hull Fo a plana point set S, S = n, the andomized incemental constuction adds the points in andom ode, always maintaining the convex hull of the points added so fa When a point 4

5 p is added, it has to locate itself, ie it has to know whethe it is outside the cuent convex hull, and in this case identify some hull edge e visible fom p As it tuns out, the amotized expected cost fo doing this in the -th step afte which the points added so fa fom a andom sample of size is popotional to a /, whee a := E#{p S \ p conv} The tick now is to expess this in tems of anothe quantity b := E#{p p vetex of conv} The sampling lemma with φ = conv then shows that a = b +1 n Fo this, we need the obsevation that p conv is equivalent to conv {s} conv, which in tun means that p is a vetex of conv {s} The expected oveall location cost which dominates the untime is then popotional to n =1 a n n b =1 Because b , this gives an On log n algoithm Howeve, the bound is much bette in some cases Fo example, if the input points ae chosen andomly fom the unit squae unit disk, espectively, we get b = Olog b = O 3, espectively [28, 20] In both cases, the algoithm actually uns in linea time In highe dimensions, an analysis along these lines is available, but equies substantial efinements [9, 30] Minimum spanning foests Let G = V, E be an edge-weighted gaph, V = n Fo D E, let msfd denote the minimum spanning foest of the gaph V, D which we assume to be unique fo all D An edge e E is called D-light if it eithe connects two components of msfd, o it has smalle weight than some edge on the unique path in msfd between its two vetices The expected linea-time algoithm fo computing msfe due to Kage, Klein and Tajan [21, 25] elies among othe insights on the following fact: if D is a andom p-sample, the expected numbe of D-light edges is bounded by n/p Using the p-sampling Lemma 12, this fact is easily deived Namely, it is a simple obsevation that e is D-light if and only if msfd msfd {e} With φd = msfd, this means that the set of D-light edges is exactly the set of violatos of D By the p-sampling lemma, if D is a andom p-sample, thei expected numbe is given by v p = 1 p p xp xp p It emains to obseve that x p n 1, because XD contains exactly the edges in msfd, fo all D Along these lines, Chan has poved a bound fo the expected numbe of D-light edges in the case whee D is a andom sample of size [4] His agument uses backwads analysis and boils down to a poof of the Sampling Lemma 11 in this specific scenaio 5

6 Backwads analysis and configuation spaces The Sampling Lemma 11 in its full geneality can easily be poved using backwads analysis, and as indicated in the pevious subsection, this is usually the way its specializations ae deived in the applications Fo this, one consides the andomized incemental constuction of φs, via adding the elements of S in andom ode, and analyzes the situation in step + 1 [30] Thee is also a connection to configuation spaces In geneal, such a space consists of an abstact set of configuations ove some set S, whee each configuation has a defining set D S and a conflict set K S is active wt S if and only if D and K S \ The goal is to compute the configuations active wt S, by adding the elements in andom ode, always maintaining the active configuations of the cuent subset The abstact famewok povides bounds fo the expected oveall stuctual change numbe of configuations eve becoming active duing that constuction [9, 27, 11] In ou case, evey subset has exactly one active configuation = φ associated with it, whee D = X and K = V 2 In this case, the sampling lemma povides a bound fo the expected stuctual change v /n that occus in step + 1 Fo example, it specializes to Theoem 914 of [11] if x +1 is bounded by a constant d In the following, we will be inteested not only in the expectation, but also in the distibution of the andom vaiable V, something the configuation space famewok does not handle Fo this, we concentate on the case in which S, φ has the stuctue of an LP-type poblem This situation coves many impotant optimization poblems, including linea pogamming and all motivating examples discussed above 3 LP-type poblems If φ maps subsets to some odeed set O, we can conside functions φ that ae monotone, ie φf φg fo F G In this situation, we can egad a pai S, φ as an optimization poblem ove O, as follows: S is an abstact set of constaints, and fo any S, φ epesents the minimum value in O subject to the constaints in The examples above ae all of this type, if we define appopiate odeings on the φ-values Fo φ = min in case of keys, we simply take the deceasing ode on keys Fo S a point set and φ = ball, we can ode the balls accoding to thei adii, while fo φ = conv, we may use the aea of conv Moeove, in all these examples, φ has anothe special popety which we efe to as locality We say that φ is local if Q and φ = φq implies V = V Q, fo all, Q S An example fo a non-local poblem is the diamete: fo a set S of points and S, we define φ to be the euclidean diamete of In Figue 2 we have φ = φq fo = {q, s} and Q = {p, q, s}, but = V V Q = {} Still, locality is pesent in many poblems of pactical elevance, the most pominent one being linea pogamming LP In a geometic fomulation of LP, S is a set of halfspaces in d-dimensional space, and φ is the lexicogaphically smallest point among all the ones that minimize some fixed linea function ove the intesection of all halfspaces in If that intesection is empty, we set φ =, with the undestanding that this value dominates all othe values If the function is unbounded ove the intesection, we set φ =, standing fo undefined 2 Some cae is in ode hee; in degeneate situations, can define seveal configuations with diffeent sets D, in which case X is the intesection of all those sets 6

7 s p >D D q Figue 2: The diamete poblem: locality fails Linea pogamming is also the motivating example fo the following definition [32] Definition 31 Let S be a finite set, O some odeed set, and φ : 2 S O { } a function, whee is assumed to be the minimum value in O { } The pai S, φ is called an LP-type poblem, if φ is monotone and local, ie if fo all Q S with φ i φ φq, and ii φ = φq implies V = V Q The concept of LP-type poblems has poved useful in the undestanding of geometic optimization, see fo example [2] Fo many poblems including linea pogamming and smallest enclosing ball, the cuently best theoetical untime bounds in the unit cost model can be obtained by an algoithm that woks fo geneal LP-type poblems [16, 23] We ecall the following futhe notations only biefly and efe to the above liteatue fo details Definition 32 Let L = S, φ be an LP-type poblem i A basis of S is an inclusion-minimal subset B with φb = φ A basis in L is a basis of some set S A basis in is a basis in L contained in ii The combinatoial dimension of L, denoted by δ = δl, is the size of a lagest basis in L iii L is egula if all bases of sets, δ egula bases have size exactly δ iv L is nondegeneate if evey set, δ has a unique basis B The following implications can easily be deived Fact 33 Let L = S, φ be an LP-type poblem and S with φ Then i φ = φs \ V, and ii the set X of exteme elements of is the intesection of all bases of If L has combinatoial dimension δ, it follows that X δ fo all, so that the sampling lemma yields v δ n + 1 7

8 In paticula, a andom sample of size δn has no moe than violatos on aveage, and this is the balancing that will pove useful below In the next section we deive bounds fo egula, nondegeneate LP-type poblems that apply to the geneal case only in a weake fom While egulaity can be enfoced in the nondegeneate case we descibe a well-behaved egulaizing constuction below, nondegeneacy is a moe subtle issue It is not known how to make a geneal LP-type poblem nondegeneate without substantially changing its stuctue [22] Fo most geometic LP-type poblems, howeve, a slight petubation of the input will entail a nondegeneate poblem, essentially equivalent to the oiginal one Most notably, this is the case fo LP Enfocing egulaity Given a nondegeneate LP-type poblem S, φ of combinatoial dimension δ, the idea is to make it egula by pumping up bases which ae too small Fo this, we define an abitay linea ode on S, and conside the function φ := φ, E, whee E consists of the vecto of the m lagest elements in \ B, fo m = minδ, B φ -values ae compaed lexicogaphically, ie by the φ-component fist If the φ-values ae equal, the lexicogaphic ode of the E-components well defined wt the chosen ode on S decides the compaison φ can be consideed as a efinement of φ Lemma 34 Matoušek [22] If L = S, φ is nondegeneate, then S, φ is a egula, nondegeneate LP-type poblem of combinatoial dimension δl Moeove, if V and V denote the violating sets of S wt φ and φ, we have the following simple but impotant fact V V 5 This holds because φ {s} > φ implies φ {s} > φ It follows that when we develop tail estimates fo the expected size of V moe geneally, fo any egula and nondegeneate LP-type poblem, those estimates then also apply to non-egula poblems 4 Tail estimates In the following, we conside egula and nondegeneate LP-type poblems S, φ with S = n and δs, φ = d, whee we assume n and d to be fixed fo the est of this section Fo given paametes d and k, we want to bound pobv k The most impotant obsevation is that this quantity does not depend on the LP-type poblem, but is meely a function of the paametes n, d, and k This follows fom a esult fist poved by Clakson [7] in the context of linea pogamming, and late genealized to LP-type poblems by Matoušek [22] Let us edeive the statement hee 8

9 Theoem 41 Let S, φ be a egula, nondegeneate LP-type poblem with S = n and δs, φ = d Then pobv = k = k+d 1 n d k d 1 d n Poof: A basis B is the basis of a set if and only if B S \ V B This means, fo any egula basis B with k violatos, thee ae n d k d sets of size which have B as thei unique basis It follows that n d k d pobv = k = b k n, = d,, n, whee b k is the numbe of egula bases with k violatos in S, φ By summing ove all k, we get n d n n d k = b k, = d,, n 6 d k=0 This system of linea equations can be witten in the fom n n n,,, = b n d, b n d 1,, b 0 T, d d + 1 n whee T is an uppe-tiangula matix with all diagonal enties equal to 1, theefoe invetible This means, the b k ae uniquely detemined by the system 6, fom which k + d 1 b k = d 1 follows via a standad binomial coefficient identity [17, equation 526] This poves the statement of the theoem This esult leads to an explicit fomula fo pobv k, but useful tail estimates do not yet follow fom that By sevee ginding it might be possible to extact good bounds diectly fom the fomula we didn t succeed, but thee is anothe appoach: as we know that the quantity in question does not depend on the paticula LP-type poblem, we might as well use ou favoite LP-type poblem in the analysis In fact, fo any given paametes n and d, thee is a canonical LP-type poblem fom which statements about the distibution of V can be extacted without pain The d-smallest numbe poblem Let N be the set {1,, n} Fo N, define min d as the d-smallest numbe in equivalently, the element of ank d in If < d, this is undefined, and min d := We have the following easy facts poofs omitted Lemma 42 i N, φ with φ := min d is a egula, nondegeneate LP-type poblem of combinatoial dimension d, if φ-values ae compaed accoding to deceasing ode in N ii The basis of any set, d, consists of the d smallest numbes in 9

10 iii s S \ violates if and only if s is smalle than the d-smallest numbe in Fo d = 1, we have min d = min, thus we ecove the LP-type poblem undelying the efficient minimum seach in a soted compact list descibed in the intoduction As a wam-up execise, let us edeive the fomula fo the numbe of bases with exactly k violatos in a egula and nondegeneate LP-type poblem, by using the fact that this numbe does not depend on the actual LP-type poblem, see Theoem 41 Obsevation 43 The d-smallest numbe poblem has k + d 1 b k = d 1 egula bases with exactly k violatos Poof: Any set B with d elements is a egula basis B has k violatos if and only if the d-smallest numbe x in B is the k + d-smallest numbe in N The elements in B \ {x} can be any d 1 among the k + d 1 smalle numbes in N The poof of this obsevation might be somewhat simple than the one we had in the geneal case, but it does not lead to new insights Howeve, the next theoem about highe moments of V is an example of a statement which we think is not immediate to pove let alone, discove without making use of the d-smallest numbe poblem Theoem 44 Let S, φ be a egula, nondegeneate LP-type poblem, a andom sample of size Fo j {0,, n }, we have E n V +j = j j+d 1 j n Poof: We evaluate the expectation fo the d-smallest numbe poblem and then use Theoem 41 Fo this, we need to count the expected numbe of sets J, J = j with J V Obseve that this inclusion holds if and only if all elements of J ae smalle than the d-smallest numbe in, equivalently, if J is among the j + d 1 smallest numbes in J Fo any set L of size + j, thee ae j+d 1 j pais, J, J = L, with this popety Thus we get n E V j = = = = J S\ J =j L =+j n + j [J V ] j + d 1 j j + d 1 j When applied to j = 2, the Theoem can be used to compute the vaiance of V, leading to a Chebyshev-type tail estimate The highe moments give still bette bounds We ae going fo Chenoff-type bounds, by exploiting the special stuctue of the d-smallest numbe poblem 10

11 A Chenoff-type tail estimate To choose a andom subset N of size, one can poceed in ounds, whee ound i selects an element s i unifomly at andom among the ones not chosen so fa Equivalently, one may choose a ank l i unifomly at andom in {1,, n + 1 i} and let s i be the element of ank l i among the ones not chosen so fa Fix some positive intege k and let U k be the andom vaiable fo the numbe of indices i with l i k We have the following elation to the andom vaiable V Lemma 45 Let = l denote the set detemined by l = l 1,, l Then U k l d V k 1 Poof: We claim that U k d implies min d k + d 1 Because the latte is equivalent to V k 1, the lemma follows To pove the claim, we fist note that s i = l i + #{j < i s j < s i } 7 Conside some set I of d indices i such that l i k fo i I Such a set exists if U k d If s i k + d 1 fo all i I, we get min d k + d 1, as equied Othewise, thee is some i I such that s i = k + e, e d Then we get #{j < i s j < k + e} = k + e l i e, which implies #{j < i s j < k + d} d As befoe, this means that min d k + d 1 Coollay 46 pobv k pobu k d 1 Chenoff-type bounds fo U k ae easy to obtain now U k can be expessed as the sum of independent andom vaiables U k,i, i = 1,,, whee and it holds that U k,i := pobu k,i = 1 = { 1, if li k 0, othewise, k n + 1 i =: p i The following is one of the basic Chenoff bounds [19] Lemma 47 With EU k = p p / and t 0, pobu k EU k t exp t2 2EU k Using t = EU k d + 1 which is nonnegative fo the values of k we will be inteested in below, we obtain pobu k d 1 exp EU k d EU k 11

12 Fix some value λ 0 and choose k in such a way that EU k = 1 + λd Then we get λd + 12 pobu k d 1 exp 21 + λd exp λ λ d The value of k that entails EU k = 1 + λd satisfies and we obtain ou esult k = 1 + λd 1 i=0 1/n i 1 + λdn, Theoem 48 Let L = S, φ be a nondegeneate LP-type poblem with S = n and dims, φ = d Fo d and any λ 0, pob V 1 + λd n exp λ λ d We have deived this bound only fo egula poblems, but as we have shown befoe, any poblem can be egulaized, and by 5, the estimate then also holds fo nonegula poblems Because EV dn / + 1 dn/, this bound establishes estimates fo the tail to the ight of the expectation It might seem that the bound is athe weak, in paticula because it does not depend on n and Howeve, it is essentially best possible, as the following lowe bound shows the actual fomulation has been chosen in ode to minimize the computational effot Theoem 49 Let L = S, φ be a nondegeneate LP-type poblem with S = n and dims, φ = d Fo d and any λ 0 such that 1 + λd /2, pob V > 1 + λd n + 1 d exp 1 + λd 1 + λ2 d 2 Poof: With U k as defined above and = l, the elation 7 immediately entails V k d min d k U k l d, so that we get pobv > k d pobu k d 1 Futhemoe, k pobu k d 1 pobu k = 0 = 1 n + 1 i i=1 1 k n + 1 With k = 1 + λdn + 1 /, it follows that 1 + λd pobu k d 1 1 exp 1 + λd 1 + λ2 d 2, using the inequality 1 x exp x x 2 fo x 1/2 An open question is whethe the statement of Theoem 48 also holds in the degeneate case It is tempting to conjectue that pobv k is maximized fo nondegeneate poblems this would yield Theoem 48 fo the geneal case Moeove, while the bound is tight in the egula case, one might be able to impove it fo a given nonegula poblem We conclude this section by poving a weake tail estimate which applies to the geneal case Using this, we can show that the numbe of violatos exceeds the expected value by no moe than a logaithmic facto, with high pobability 12

13 Theoem 410 Let L = S, φ be an LP-type poblem with S = n and dims, φ = d Fo d and any λ 0, pob V ln ne d + λ d n exp λd Poof: Let B k denote the set of egula bases with exactly k violatos ecall that a egula basis is a basis of some set with d Any fixed B B k is a basis of all the sets satisfying B S \ V B It follows that B is a basis of a andom sample of size with pobability n B k B n n k n We have V = k if and only if has some basis equivalently, all its bases in B k, which gives n k pobv = k b k n, b k = B k Consequently, whee we know that n n l pobv k b l n, n b l l=k l=k l=k n := d d i=0 n, i because all bases have size at most d Then we can futhe ague that n n l n pobv k b l max n l=kn d Since cf [24] and n k n = n d ne d d n k n 1 k 1 k k 1 1 k, n n 1 n + 1 n we finally get, by substituting k = ln ne d + λ d n, ne d ln ne d pobv k 1 + λ d ne d exp ln ne d d d + λ d = exp λd 5 Multiple Picing and Clakson s eduction Scheme The simplex method [5] is usually the most efficient algoithm to solve linea pogamming poblems in pactice Even in the theoetical setting, all known algoithms to solve geneal LP-type poblems boil down to vaiants of the dual simplex method, when they ae applied to LP [13] In this section we intoduce and analyze an algoithm in the geneal famewok, 13

14 which although being new in its pecise fomulation follows a well-known design paadigm, whose simplex countepat is known as multiple picing[5] The idea of multiple picing is to educe a lage poblem to a hopefully small numbe of small poblems This can be useful in case the whole poblem does not fit into main memoy, but it also helps in geneal to educe the cost of a single simplex iteation Taking a slightly diffeent appoach, patial picing[5] is a elated technique following the same paadigm Applications have been found in the context of vey lage-scale linea pogamming [3], but also in geometic optimization [14, 15] We will not elaboate on those simplex techniques hee; the eade may veify that the algoithm we ae going to pesent is actually a vaiant of multiple picing, when tanslated into simplex teminology Conside an LP-type poblem S, φ not necessaily nondegeneate of combinatoial dimension d, and assume we ae given an algoithm lp typeg, B to compute fo any subset G of S some basis B G of G, given a candidate basis B G Of couse, one can diectly solve the poblem of finding B S by calling lp type with the lage set S and some basis B S As we will see, an efficient altenative is povided by the following method, paameteized with a sample size We assume the initial basis B to be fixed fo the est of this section Algoithm 51 lp type sampling S, B: * etuns some basis B S of S * choose with =, S \ B at andom G := B EPEAT B :=lp typeg, B G := G V B UNTIL V B = ETUN B lp type sampling educes the poblem to seveal calls of lp type, and Fact 33i shows that if the pocedue teminates, V B = implies that B is a basis of S Moeove, it must eventually teminate, because evey ound adds at least one element to G The algoithm captues the spiit of Clakson s linea pogamming algoithm [8] and its genealizations [1, 16], but is simple and moe pactical To guaantee its theoetical complexity, Clakson s algoithm daws a andom sample in evey ound, and it estats a ound wheneve V B tuns out to be too lage Thus, Algoithm 51 can be intepeted as the canonical simplification of Clakson s algoithm fo pactical use, whee one obseves that esampling and estating ae not necessay and even decease the efficiency The geneal phenomenon behind this is that often the theoetically best algoithms ae not competitive in pactice, while the algoithms one actually chooses in an implementation cannot be analyzed On the one hand this is due to the fact that the wost-case complexity is an inappopiate measue in many pactical situations; on the othe hand, sometimes algoithms used in pactice ae simply not undestood, although they might allow a wost-case analysis In case of Algoithm 51, we have the fotunate situation that it combines efficiency in pactice with povable time bounds developed below With the pocedue lp type eplaced by a call to a standad simplex implementation, the method has been successfully used in a linea pogamming code fo geometic optimization [14, 15], without any futhe changes In its oiginal vesion, due to Clakson, Algoithm 51 is a building-block of an ingenious linea-time algoithm fo linea pogamming in constant dimension d [8, 16] 14

15 The theoetical analysis stats with a bound on the numbe of ounds Obsevation 52 Clakson[8] Fix some basis B S of S Then in evey ound except the last one, V B contains an element of B S In paticula, thee ae at most d + 1 ounds Poof: Assume that B S is disjoint fom V B Fom Fact 33 and monotonicity we then get φb = φs \ V B φb S = φs, fom which φb = φs follows Locality then implies V B = V S =, which means that we ae aleady in the last ound The citical paamete we will be inteested in is the size of G in the last ound If this is small, then all calls to lp typeg, B ae cheap Let us fix some notation fo that We define S := S \ B, B the initial candidate basis plugged into lp type sampling By, V i, and Gi B i we denote the sets B, V B and G computed in ound i Futhemoe, we set G 0 = B, while B 0 0 and V ae undefined This means, we have B i is a basis of Gi 1, V i = V Gi 1 If the algoithm pefoms exactly l ounds, sets with indices i > l ae defined to be the coesponding sets in ound l We will need a genealization of Obsevation 52 Lemma 53 Fo j < i l, B i V j Poof: Assume on the contay that B i 33 and monotonicity then imply φg j 1 V j = As in the poof of Obsevation 52, Fact = φs \ V j φbi = φgi 1, a contadiction to the fact that φg stictly inceases in evey but the last ound The following lemma is the cucial esult It intepets Algoithm 51 as an LP-type poblem itself! Unde this intepetation, the set G in the last ound is essentially the set of violatos of the initial sample Then, the techniques of the pevious sections the sampling lemma, and the tail estimates can be applied to bound the expected size of G, and even get Chenoff-type bounds fo the distibution of G Lemma 54 Fo S := S \ B define φ = φg 0, φg1,, φgd 1 Then the following holds i S, φ is an LP-type poblem of combinatoial dimension at most d+1 2, unde the lexicogaphic ode of the d-tuples φ ii The set V := {s S \ φ φ {s}} of violatos of wt φ is given by V = V 1 V d 15 = Gd \ B

16 iii If S, φ is nondegeneate, so is S, φ Befoe we go into the technical although not difficult poof, let us deive the main esult of this section, namely the analysis of Algoithm 51 This analysis is now meely a consequence of pevious esults Theoem 55 Fo S a andom sample of size, E G d d + 1 n d + + d Choosing = d n/2 yields E G d 2d + 1 n 2 Poof: The fist inequality diectly follows fom the sampling lemma, applied to the LP-type poblem S, φ, togethe with pat ii of the pevious lemma The second inequality is outine The theoem shows that Algoithm lp type sampling educes a poblem of size n to at most d poblems of expected size no moe than Od n This explains the pactical efficiency of multiple picing and simila eduction schemes if d n If S, φ is nondegeneate, we get the following tail estimate, using pat iii of Lemma 54 and Theoem 48 Again, outine computations yield Theoem 56 If S, φ is a nondegeneate LP-type poblem, then fo S a andom sample of size = d n/2, and λ 0, pob G d n 2 + λd + 1 exp λ2 d λ 2 In the degeneate case, Theoem 410 can be used to deive the following weake but still useful esult Theoem 57 If S, φ is a geneal LP-type poblem, then fo S a andom sample of size = d n ln n/2, and λ 0, pob G d n ln n d λ d + 1 exp λ 2 2 We conclude this section with the poof of Lemma 54 Poof: We stat by establishing an auxiliay claim: fo any set Q with Q = T S and i < d, φg j = φgj Q, j i implies G j Q = G j T, j i + 1, V j+1 Q = V j+1, j i To pove the claim, we poceed by induction on i, noting that the statements hold fo i = 0 by locality of φ Now assume the implications ae tue fo j i 1 Then we get G i Q = Gi 1 Q = G i 1 V i Q T V i = Gi 16 T

17 Because φg i = φgi Q, locality of φ implies V i+1 Q = V i+1, which in tun poves G i+1 Q = G i+1 T This establishes the claim To poceed, let us fist pove pat ii of Lemma 54 Assume s V, set Q := {s} and conside the lagest index i < d 1 such that By the claim above, G i+1 Q = G i+1 φg j = φgj Q, j i {s}, and monotonicity of φ implies φg i+1 < φg i+1 Q 8 This means, s V i+2 On the othe hand, if s V, then the pecondition of the claim holds fo i = d 1, implying V j+1 = V j+1 Q s, j d 1 This means s V 1,, V d To pove i, we need to veify monotonicity and locality cf Definition 31 Inequality 8 shows that φ φ {s} in the lexicogaphic ode, fo all s V, and this implies monotonicity Fo locality, assume Q with φ = φ Q Fom the claim and pat ii, we get V = d i=1 V i = d i=1 V i Q = V Q, and this is the equied popety It emains to bound the combinatoial dimension of S, φ φ B = φ, fo d B := B i i=1 To this end we pove that We equivalently show that φg j = φgj B, fo j d 1, using induction on j Fo j = 0, we get G j = Gj B \ B, hence φg j = φgj B \ B = φg j B, 9 because \ B is disjoint fom B 1, the basis of G0 Hence, \ B can be emoved fom G 0 without changing the φ-value Now assume the statement holds fo j d 2 and conside the case j = d 1 By the claim, we get G j = Gj B \ B, so as befoe, 9 follows, because \ B is disjoint fom the basis B j+1 of G j To bound the size of B, we obseve that B i d + 1 i, 17

18 fo all i l the numbe of ounds in which V B This follows fom Lemma 53: B i has at least one element in each of the i 1 sets V 1,, V i 1, which ae in tun disjoint fom Hence we get l B B i d i=1 iii Nondegeneacy of S, φ follows, if we can show that evey set S has the set B as its unique basis To this end we pove that wheneve we have L with φ L = φ, then B L Fix L with φ L = φ, ie By the claim, this implies φg i G i = φgi L, i d 1 = Gi L and the nondegeneacy of φ yields that G i i It follows that G d 1 L contains so L contains L d i=1 \ L, i d 1, and Gi L d i=1 B i B i, = d have the same unique basis Bi+1, fo all i=1 B i The latte equality holds because \L is disjoint fom G d 1 L, thus in paticula fom the union of the B i 6 Conclusion The cuious fact that in the egula and nondegeneate case the distibution of V does not depend on the actual LP-type poblem, deseves a wod of waning: namely, this popety does not mean that all nondegeneate LP-type poblems with given paametes n and d ae equally difficult o easy to solve On the contay, because the andom vaiable V does not depend on the actual poblem, it does not cay any infomation about the difficulty of a paticula poblem Thee ae vey easy poblems like d-smallest numbe, and vey difficult ones like linea pogamming Fo example, Algoithm 51 neve needs moe than two ounds in case of d-smallest numbe, and fo othe easy LP-type poblems chaacteized by the following popety: fo any sets B such that φb = φ, and fo any set T, φb T = φ T holds This means, elements in \ B can be fogotten, as they will not contibute to the final solution The absence of this popety is what makes linea pogamming and othe poblems difficult In geneal, it seems that the combinatoial dimension of the LP-type poblem S, φ deived fom S, φ accoding to the definition in Lemma 54 is a moe meaningful indicato of S, φ s difficulty then δs, φ itself Fo example, in case of d-smallest numbe, we get δs, φ = d, 18

19 much less than the Od 2 uppe bound This altenative notion of dimension needs to be futhe investigated An open poblem that emains is to impove the tail estimates in case of degeneate LP-type poblems Hee, the distibution of V typically depends on the concete instance, and so does b k, the numbe of bases with k violatos Using only tivial bounds fo the numbes b k, we have obtained the weake estimate given by Theoem 410, indicating that this estimate might not be the final answe Acknowledgment We like to thank the efeee fo caefully pointing out simplifications and suggesting impovements in the pesentation In paticula, we ae gateful fo the question concening the shapness of ou main Chenoff-type bound efeences [1] I Adle and Shami A andomized scheme fo speeding up algoithms fo linea and convex pogamming with high constaints-to-vaiable atio Math Pogamming, 61:39 52, 1993 [2] N Amenta Helly-type theoems and genealized linea pogamming Discete Comput Geom, 12: , 1994 [3] E Bixby, J W Gegoy, I J Lustig, E Masten, and D F Shanno Vey lagescale linea pogamming: a case study in combining inteio point and simplex methods Opeations eseach, 405: , 1992 [4] T Chan Backwads analysis of the Kage-Klein-Tajan algoithm fo minimum spanning tees Infom Poc Lettes, 67: , 1998 [5] V Chvátal Linea Pogamming W H Feeman, New Yok, NY, 1983 [6] K L Clakson New applications of andom sampling in computational geomety Discete Comput Geom, 2: , 1987 [7] K L Clakson A bound on local minima of aangements that implies the uppe bound theoem Discete Comput Geom, 10: , 1993 [8] K L Clakson Las Vegas algoithms fo linea and intege pogamming J ACM, 42: , 1995 [9] K L Clakson and P W Sho Applications of andom sampling in computational geomety, II Discete Comput Geom, 4: , 1989 [10] T H Comen, C E Leiseson, and L ivest Intoduction to Algoithms The MIT Pess, Cambidge, MA, 1990 [11] M de Beg, M van Keveld, M Ovemas, and O Schwazkopf Computational Geomety: Algoithms and Applications Spinge-Velag, Belin, 1997 [12] D Dubhashi and D anjan Geate expectations BICS Newslette, 5,

20 [13] B Gätne andomized Optimization by Simplex-Type Methods PhD thesis, Feie Univesität Belin, 1995 [14] B Gätne Exact aithmetic at low cost a case study in linea pogamming Computational Geomety - Theoy and Applications, 13: , 1999 [15] B Gätne and S Schönhe An efficient, exact and geneic quadatic pogamming solve fo geometic optimization In Poc 16th ACM Sympos Comput Geom, 2000 [16] B Gätne and E Welzl Linea pogamming andomization and abstact famewoks In Poc 13th Sympos Theoet Aspects Comput Sci, volume 1046 of Lectue Notes Comput Sci, pages Spinge-Velag, 1996 [17] L Gaham, D E Knuth, and O Patashnik Concete Mathematics Addison-Wesley, eading, MA, 1989 [18] L J Guibas, D E Knuth, and M Shai andomized incemental constuction of Delaunay and Voonoi diagams Algoithmica, 7: , 1992 [19] T Hageup and C üb A guided tou of Chenoff bounds Infom Pocess Lett, 33: , 1990 [20] S Ha-Peled On the expected complexity of andom convex hulls Technical epot 330, School of Math Sciences, Tel-Aviv Univesity, 1998 [21] D Kage, P N Klein, and E Tajan A andomized linea-time algoithm to find minimum spanning tees J ACM, 42: , 1995 [22] J Matoušek On geometic optimization with few violated constaints Discete Comput Geom, 14: , 1995 [23] J Matoušek, Micha Shai, and Emo Welzl A subexponential bound fo linea pogamming Algoithmica, 16: , 1996 [24] J Matoušek and J Nešetřil Invitation to Discete Mathematics Oxfod Univesity Pess, 1998 [25] Motwani and P aghavan andomized Algoithms Cambidge Univesity Pess, New Yok, NY, 1995 [26] K Mulmuley A fast plana patition algoithm, I J Symbolic Comput, 103-4: , 1990 [27] K Mulmuley Computational Geomety: An Intoduction Though andomized Algoithms Pentice Hall, Englewood Cliffs, NJ, 1994 [28] A ényi and Sulanke Übe die konvexe Hülle von n zufällig gewählten Punkten Z Wahscheinlichkeitstheoie, 2:75 84, 1963 [29] Seidel Small-dimensional linea pogamming and convex hulls made easy Discete Comput Geom, 6: , 1991 [30] Seidel Backwads analysis of andomized geometic algoithms In J Pach, edito, New Tends in Discete and Computational Geomety, volume 10 of Algoithms and Combinatoics, pages Spinge-Velag,

21 [31] Seidel Pesonal communication, 1996 [32] M Shai and E Welzl A combinatoial bound fo linea pogamming and elated poblems In Poc 9th Sympos Theoet Aspects Comput Sci, volume 577 of Lectue Notes Comput Sci, pages Spinge-Velag, 1992 [33] E Welzl Smallest enclosing disks balls and ellipsoids In H Maue, edito, New esults and New Tends in Compute Science, volume 555 of Lectue Notes Comput Sci, pages Spinge-Velag,