Output-Sensitive Algorithms for Computing Nearest-Neighbour Decision Boundaries

Transcription

1 Output-Sensitive Algoithms fo Computing Neaest-Neighou Decision Boundaies David Bemne Eik Demaine Jeff Eickson John Iacono Stefan Langeman Pat Moin Godfied Toussaint ABSTRACT. Given a set R of ed points and a set B of lue points, the neaest-neighou decision ule classifies a new point q as ed (espectively, lue) if the closest point to q in R B comes fom R (espectively, B). This ule implicitly patitions space into a ed set and a lue set that ae sepaated y a ed-lue decision ounday. In this pape we develop output-sensitive algoithms fo computing this decision ounday fo point sets on the line and in R 2. Both algoithms un in time O(n log k), whee k is the nume of points that contiute to the decision ounday. This unning time is the est possile when paameteizing with espect to n and k. 1 Intoduction Let S e a set of n points in the plane that is patitioned into a set of ed points denoted y R and a set of lue points denoted y B. The neaest-neighou decision ule classifies a new point q as the colo of the closest point to q in S. The neaest-neighou decision ule is popula in patten ecognition as a means of leaning y example. Fo this eason, the set S is often efeed to as a taining set. Seveal popeties make the neaest-neighou decision ule quite attactive, including its intuitive simplicity and the theoem that the asymptotic eo ate of the neaest-neighou ule is ounded fom aove y twice the Bayes eo ate [6, 8, 15]. (See [16] fo an extensive suvey of the neaestneighou decision ule and its elatives.) Futhemoe, fo point sets in small dimensions, thee ae efficient and pactical algoithms fo pepocessing a set S so that the neaest neighou of a quey point q can e found quickly. The neaest-neighou decision ule implicitly patitions the plane into a ed set and a lue set that meet at a ed-lue decision ounday. One attactive aspect of the neaest-neighou decision ule is that it is often possile to educe the size of the taining set S without changing the decision ounday. To see this, conside the Voonoĭ diagam of S, which patitions the plane into convex (possily unounded) polygonal Voonoĭ cells, whee the Voonoĭ cell of point p S is the set of all points that ae close to p than to any othe point in S (see Figue 1.a). If the Voonoĭ cell of a ed point is completely suounded y the Voonoi cells of othe ed points then the point can e emoved fom S and this will not change This eseach was patly funded y the Alexande von Humoldt Foundation and NSERC Canada Faculty of Compute Science, Univesity of New Bunswick, emne@un.ca MIT Laoatoy fo Compute Science, edemaine@mit.edu Compute Science Depatment, Univesity of Illinois, jeffe@cs.uiuc.edu Polytechnic Univesity, jiacono@poly.edu Chagé de echeches du FNRS, Univesité Lie de Buxelles, stefan.langeman@ul.ac.e School of Compute Science, Caleton Univesity, moin@cs.caleton.ca School of Compute Science, McGill Univesity, godfied@cs.mcgill.ca 1

2 (a) () Figue 1: The Voonoĭ diagam (a) efoe Voonoĭ condensing and () afte Voonoĭ condensing. Note that the decision ounday (in old) is unaffected y Voonoĭ condensing. Note: In this figue, and all othe figues, ed points ae denoted y white cicles and lue points ae denoted y lack disks. the classification of any point in the plane (see Figue 1.). We say that these points do not contiute to the decision ounday, and the emaining points contiute to the decision ounday. The peceding discussion suggests that one appoach to educing the size of the taining set S is to simply compute the Voonoĭ diagam of S and emove any points of S whose Voonoĭ cells ae suounded y Voonoĭ cells of the same colo. Indeed, this method is efeed to as Voonoĭ condensing [17]. Thee ae seveal O(n log n) time algoithms fo computing the Voonoĭ diagam a set of points in the plane, so Voonoĭ condensing can e implemented to un in O(n log n) time. 1 Howeve, in this pape we show that we can do significantly ette when the nume of points that contiute to the decision ounday is small. Indeed, we show how to do Voonoĭ condensing in O(n log k) time, whee k is the nume of points that contiute to the decision ounday (i.e., the nume of points of S that emain afte Voonoĭ condensing). Algoithms, like these, in which the size of the input and the size of the output play a ole in the unning time ae efeed to as output-sensitive algoithms. Reades familia with the liteatue on output-sensitive convex hull algoithms may ecognize the expession O(n log k) as the unning time of optimal algoithms fo computing convex hulls of n point sets with k exteme points, in 2 o 3 dimensions [2, 4, 5, 12, 18]. This is no coincidence. Given a set of n points in R 2, we can colo them all ed and add thee lue points at infinity (see Figue 2). In this set, the only points that contiute to the neaest-neighou decision ounday ae the thee lue points and the ed points on the convex hull of the oiginal set. Thus, identifying the points that contiute to the neaest-neighou decision ounday is at least as difficult as computing the exteme points of a set. The emainde of this pape is oganized as follows: In Section 2 we descie an algoithm fo computing the neaest-neighou decision ounday of points on a line that uns in O(n log k) time. In Section 3 we pesent an algoithm fo points in the plane that also uns in O(n log k) time. Finally, in Section 4 we summaize and conclude with open polems. 1 Histoically, the fist efficient algoithm fo specifically computing the neaest-neighou decision ounday is due to Dasaathy and White [7] and uns in O(n 4 ) time. The fist O(n log n) time algoithm fo computing the Voonoĭ diagam of a set of n points in the plane is due to Shamos [14]. 2

3 Figue 2: The elationship etween convex hulls and decision oundaies. Each vetex of the convex hull of R contiutes to the decision ounday. 2 A 1-Dimensional Algoithm In the 1-dimensional vesion of the neaest-neighou decision ounday polem, the input set S consists of n eal numes. Imagine soting S, so that S = {s 1,..., s n } whee s i < s i+1 fo all 1 i < n. The decision ounday consists of all pais (s i, s i+1 ) whee s i is ed and s i+1 is lue, o vice-vesa. Thus, this polem is solveale in linea-time if the points of S ae soted. Since soting the elements of S can e done using any nume of O(n log n) time soting algoithms, this immediately implies an O(n log n) time algoithm. Next, we give an algoithm that uns in O(n log k) time and is simila in spiit to Hoae s quicksot [11]. To find the decision ounday in O(n log k) time, we egin y computing the median element m = s n/2 in O(n) time using any one of the existing linea-time median finding algoithms (see [3]). Using an additional O(n) time, we split S into the sets S 1 = {s 1,..., s n/2 1 } and S 2 = {s n/2 +1,..., s n } y compaing each element of S to the median element m. At the same time we also find s n/2 1 and S n/2 +1 y finding the maximum and minimum elements of S 1 and S 2, espectively. We then check if (s n/2 1, m) and/o (m, s n/2 +1 ) ae pat of the decision ounday and epot them if necessay. At this point, a standad divide-and-conque algoithm would ecuse on oth S 1 and S 2 to give an O(n log n) time algoithm. Howeve, we can impove on this y oseving that it is not necessay to ecuse on a supolem if it contains only elements of one colo, since it will not contiute a pai to the decision ounday. Theefoe, we ecuse on each of S 1 and S 2 only if they contain at least one ed element and one lue element. 3

4 The coectness of the aove algoithm is clea. To analyze its unning time we oseve that the unning time is ounded y the ecuence T (n, k) O(n) + T (n/2, l) + T (n/2, k l), whee l is the nume of points that contiute to the decision ounday in S 1 and whee T (1, k) = O(1) and T (n, 0) = O(n). An easy inductive agument that uses the concavity of the logaithm shows that this ecuence is maximized when l = k/2, in which case the ecuence solves to O(n log k) [5]. Theoem 1. The neaest-neighou decision ounday of a set of n eal numes can e computed in O(n log k) time, whee k is the nume of elements that contiute to the decision ounday. 3 A 2-Dimensional Algoithm In the 2-dimensional neaest-neighou decision ounday polem the Voonoĭ cells of S ae (possily unounded) convex polygons and the goal is to find all Voonoĭ edges that ound two cells whose defining points have diffeent colos. Thoughout this section we will assume that the points of S ae in geneal position so that no fou points of S lie on a common cicle. This assumption is not vey estictive, since geneal position can e simulated using infinitesmal petuations of the input points. It will e moe convenient to pesent ou algoithm using the teminology of Delaunay tiangulations. A Delaunay tiangle in S is a tiangle whose vetices (v 1, v 2, v 3 ) ae in S and such that the cicle with v 1, v 2 and v 3 on its ounday does not contain any point of S in its inteio. A Delaunay tiangulation of S is a patitioning of the convex hull of S into Delaunay tiangles. Altenatively, a Delaunay edge is a line segment whose vetices (v 1, v 2 ) ae in S and such that thee exists a cicle with v 1 and v 2 on its ounday that does not contain any point of S in its inteio. When S is in geneal position, the Delaunay tiangulation of S is unique and contains all tiangles whose edges ae Delaunay edges (see [13]). It is well known that the Delaunay tiangulation and the Voonoi diagam ae dual in the sense that two points of S ae joined y an edge in the Delaunay tiangulation if and only if thei Voonoi cells shae an edge. We call a Delaunay tiangle o Delaunay edge ichomatic if its set of defining vetices contains at least one ed and at least one lue point of S. Thus, the polem of computing the neaest-neighou decision ounday is equivalent to the polem of finding all ichomatic Delaunay edges. 3.1 The High Level Algoithm In the next few sections, we will descie an algoithm that, given a value κ k, finds the set of all ichomatic Delaunay tiangles in S in O((κ 2 + n) log κ) time, which fo κ n simplifies to O(n log κ). To otain an algoithm that uns in O(n log k) time, we epeatedly guess the value of κ, un the algoithm until we find the entie decision ounday o until it detemines that κ < k and, in the latte case, estat the algoithm with a lage value of κ. If we eve each a point whee the value of κ exceeds n then we stop the entie algoithm and un an O(n log n) time algoithm to compute the entie Delaunay tiangulation of S. The values of κ that we use ae κ = 2 2i fo i = 0, 1, 2,..., log log n. Since the algoithm will 4

5 Figue 3: A pivot opeation. teminate once κ k o κ n, the total cost of all uns of the algoithm is theefoe as equied. T n,k = log log k i=0 O(n log 2 2i ) = log log k i=0 O(n2 i ) = O(n log k), 3.2 Pivots A key suoutine in ou algoithm is the pivot 2 opeation. A pivot in the set of points S takes as input a ay and epots the lagest cicle whose cente is on the ay, has the oigin of the ay on its ounday and has no point of S in its inteio. We will make use of the following data stuctuing esult, due to Chan [4]. Fo completeness, we also include a poof. Lemma 1 (Chan 1996). Let S e a set of n points in R 2. Then, fo any intege 1 m n, thee exists a data stuctue of size O(n) that can e constucted in O(n log m) time, and that can pefom pivots in S in O( n m log m) time pe pivot. Poof. Dokin and Kikpatick [9, 10] show how to pepocess a set S of n points in O(n log n) time to answe pivot queies in O(log n) time pe quey. Chan s data stuctue simply patitions S into n/m goups each of size m and then uses the Dokin-Kikpatick data stuctue on each goup. The time to uild all n/m data stuctues is n m O(m log m) = O(n log m). To pefom a quey, we simply quey each of the n/m data stuctues in O(log m) time pe data stuctue and epot the smallest cicle found, fo a quey time of n m O(log m) = O( n m log n). In the following, we will e using Lemma 1 with a value of m = κ 2, so that the time to constuct the data stuctue is O(n log κ) and the quey time is O( n κ 2 log κ). We will use two such data stuctues, one fo pefoming pivots in the set R of ed points and one fo pefoming pivots in the set B of lue points. 2 The tem pivot comes fom linea pogamming. The elationship etween a (pola dual) linea pogamming pivot and the cicula pivot descied hee is evident when we conside the paaolic lifting that tansfoms the polem of computing a 2-dimensional Delaunay tiangulation to that of computing a 3-dimensional convex hull of a set of points on the paaoloid z = x 2 + y 2. In this case, the cicle is the pojection of the intesection of a plane with the paaoloid. 5

6 C C C (a) () Figue 4: The (a) fist and () second pivot used to find a ichomatic edge (, ). 3.3 Finding the Fist Edge The fist step in ou algoithm is to find a single ichomatic edge of the Delaunay tiangulation. Refe to Figue 4. To do this, we egin y choosing any ed point and any lue point. We then pefom a pivot in the set B along the ay with oigin that contains. This gives us a cicle C that has no lue points in its inteio and has as well as some lue point (possily = ) on its ounday. Next, we pefom a pivot in the set R along the ay oiginating at and passing though the cente of C. This gives us a cicle C 1 that has no point of S in its inteio and has and some ed point (possily = ) on its ounday. Theefoe, (, ) is a ichomatic edge in the Delaunay tiangulation of S. The aove agument shows how to find a ichomatic Delaunay edge using only 2 pivots, one in R and one in B. The second pat of the agument also implies the following useful lemma. Lemma 2. If thee is a cicle with a ed point and a lue point on its ounday, and no ed (espectively, lue) points in its inteio, then (espectively, ) contiutes to the decision ounday. 3.4 Finding Moe Points Let Q e the set of points that contiute to the decision ounday, i.e., the set of points that ae the vetices of ichomatic tiangles in the Delaunay tiangulation of S. Suppose that we have aleady found a set P Q and we wish to eithe (1) find a new point p Q \ P o (2) veify that P = Q. To do this, we will make use of the augmented Delaunay tiangulation of P (see Figue 5). This is the Delaunay tiangulation of P {v 1, v 2, v 3 }, whee v 1, v 2, and v 3 ae thee lack points at infinity (see Figue 5). Fo any tiangle t, we use the notation C(t) to denote the cicle whose ounday contains the thee vetices of t (note that if t contains a lack point then C(t) is a halfplane). The following lemma allows us to tell when we have found the entie set of points Q that contiute to the decision ounday. Lemma 3. Let P Q. The following statements ae equivalent: 1. Fo evey tiangle t in the augmented Delaunay tiangulation of P, if t has a lue (espectively, ed) vetex then C(t) does not have a ed (espectively, lue) vetex of S in its inteio. 2. P = Q. 6

7 v 3 v 1 v 2 Figue 5: The augmented Delaunay tiangulation of S. t C(t) Figue 6: If Statement 1 of Lemma 3 is not tue then P Q. Poof. Fist we show that if Statement 1 of the lemma is not tue, then Statement 2 is also not tue, i.e., P Q. Suppose thee is some tiangle t in the augmented Delaunay tiangulation of P such that t has a lue vetex and C(t) contains a ed point of S in its inteio. Pivot in R along the ay oiginating at and passing though the cente of C(t) (see Figue 6). This will give a cicle C with and some ed point / P on its ounday and with no ed points in its inteio. Theefoe, y Lemma 2, contiutes to the decision ounday and is theefoe in Q, so P Q. A symmetic agument applies when t has a ed vetex and C(t) contains a lue vetex in its inteio. Next we show that if Statement 2 of the lemma is not tue then Statement 1 is not tue. Suppose that P Q. Let e a point in Q \ P and, without loss of geneality, assume is a ed point. Since is in Q, thee is a cicle C with and some othe lue point on its ounday and with no points of S in its inteio. We will use and to show that the augmented Delaunay tiangulation of P contains a tiangle t such that eithe (1) is a vetex of t and C(t) contains in its inteio, o (2) C(t) contains oth and in its inteio. In eithe case, Statement 1 of the lemma is not tue ecause of tiangle t. Refe to Figue 7 fo what follows. Conside the lagest cicle C 1 that is concentic with C and that contains no point of P in its inteio (this cicle is at least as lage as C). The cicle C 1 will have at least one point p 1 of P on its ounday (it could e that p 1 =, if P ). Next, pefom a pivot in P along the ay oiginating at p 1 and containing the cente of C 1. This will give a cicle C 2 that contains C 1 and with two points p 1 and p 2 of P {v 1, v 2, v 3 } on its ounday and with no points of P {v 1, v 2, v 3 } 7

8 in its inteio. Theefoe, (p 1, p 2 ) is an edge in the augmented Delaunay tiangulation of P. The edge (p 1, p 2 ) patitions the inteio of C 2 into two pieces, one that contains and one that does not. It is possile to move the cente of C 2 along the pependicula isecto of (p 1, p 2 ) maintaining p 1 and p 2 on the ounday of C 2. Thee ae two diections in which the cente of C 2 can e moved to accomplish this. In one diection, say d, the pat of the inteio that contains only inceases, so move the cente in this diection until a thid point p 3 P {v 1, v 2, v 3 } is on the ounday of C 2. The esulting cicle has the points p 1, p 2, and p 3 on its ounday and no points of P in its inteio, so p 1, p 2 and p 3 ae the vetices of a tiangle t in the augmented Delaunay tiangulation of P. The cicumcicle C(t) contains in its inteio and contains eithe in its inteio o on its ounday. In eithe case, t contadicts Statement 1, as pomised. Note that the fist paagaph in the poof of Lemma 3 gives a method of testing whethe P = Q, and when this is not the case, of finding a point in Q\P. Fo each tiangle t in the Delaunay tiangulation of P, if t contains a lue vetex then pefom a pivot in R along the ay oiginating at and passing though C(t). If the esult of this pivot is C(t), then do nothing. Othewise, the pivot finds a cicle C with no ed points in its inteio and that has one lue point and one ed point / P on its ounday. By Lemma 2, the point must e in Q. If t contains a ed vetex, epeat the aove pocedue swapping the oles of ed and lue. If oth pivots (fom the ed point and the lue point) find the cicle C(t), then we have veified Statement 1 of Lemma 3 fo the tiangle t. The aove pocedue pefoms at most two pivots fo each tiangle t in the augmented Delaunay tiangulation of P. Theefoe, this pocedue pefoms O( P ) = O(κ) pivots. Since we epeat this pocedue at most κ times efoe deciding that κ < k, we pefom O(κ 2 ) pivots, at a total cost of O(κ 2 n κ log κ) = O(n log κ). The only othe wok done y the algoithm is that of ecomputing 2 the augmented Delaunay tiangulation of P each time we add a new vetex to P. Since each such computation takes O( P log P ) time and P κ, the total amount of wok done in computing all these tiangulations is O(κ 2 log κ). In summay, we have an algoithm that given S and κ decides whethe the condensed set Q of points in S that contiute to the decision ounday has size at most κ, and if so, computes Q. This algoithm uns in O((κ 2 + n) log κ) time. By tying inceasingly lage values of κ as descied in Section 3.1 we otain ou main theoem. Theoem 2. The neaest-neighou decision ounday of a set of n points in R 2 can e computed in O(n log k) time, whee k is the nume of points that contiute to the decision ounday. Remak: Theoem 2 extends to the case whee thee ae moe than 2 colo classes and ou goal is to find all Voonoĭ edges ounding two cells of diffeent colo. The only modification equied is that, fo each colo class, R, we use two pivoting data stuctues, one fo R and one fo S \ R. When pefoming pivots fom a point in R, we use the data stuctue fo pivots in S \ R. Othewise, the details of the algoithm ae identical. Remak: In the patten-ecognition community patten classification ules ae often implemented as neual netwoks. In the teminology of neual netwoks, Theoem 2 states that it is possile, in O(n log k) time, to design a simple one-laye neual netwok that implements the neaest-neighou decision ule and uses only k McCulloch-Pitts neuons (theshold logic units). 8

9 C 1 C p 1 p 1 = C = C 1 p 2 p 1 p 2 p 1 = C 2 C 1 C 2 C 1 p 2 t p 1 p 2 t p 1 = p 3 p 3 C 2 C 2 C(t) C(t) (1) (2) Figue 7: If P Q then Statement 1 of Lemma 3 is not tue. The left column (1) coesponds to the case whee P and the ight column (2) coesponds to the case whee P. 9

10 4 Conclusions We have given O(n log k) time algoithms fo computing neaest-neighou decisions oundaies in 1 and 2 dimensions, whee k is the nume of points that contiute to the decision ounday. A standad application of Ben-O s lowe-ound technique [1] shows that even the 1-dimensional algoithm is optimal in the algeaic decision tee model of computation. We have not studied algoithms fo dimensions d 3. In this case, it is not even clea what the tem output-sensitive means. Should k e the nume of points that contiute to the decision ounday, o should k e the complexity of the decision ounday? In the fist case, k n fo any dimension d, while in the second case, k could e as lage as Ω(n d/2 ). To the est of ou knowledge, oth ae open polems. Refeences [1] M. Ben-O. Lowe ounds fo algeaic computation tees (peliminay epot). In Poceedings of the Fifteenth Annual ACM Symposium on Theoy of Computing, pages 80 86, [2] B. K. Bhattachaya and S. Sen. On a simple, pactical, optimal, output-sensitive andomized plana convex hull algoithm. Jounal of Algoithms, 25(1): , [3] M. Blum, R. W. Floyd, V. Patt, R. L. Rivest, and R. E. Tajan. Time ounds fo selection. Jounal of Computing and Systems Science, 7: , [4] T. M. Chan. Optimal output-sensitive convex hull algoithms in two and thee dimensions. Discete & Computational Geomety, 16: , [5] T. M. Chan, J. Snoeyink, and C. K. Yap. Pimal dividing and dual puning: Output-sensitive constuction of fou-dimensional polytopes and thee-dimensional Voonoi diagams. Discete & Computational Geomety, 18: , [6] T. M. Cove and P. E. Hat. Neaest neighou patten classification. IEEE Tansactions on Infomation Theoy, 13:21 27, [7] B. Dasaathy and L. J. White. A chaacteization of neaest-neighou ule decision sufaces and a new appoach to geneate them. Patten Recognition, 10:41 46, [8] L. Devoye. On the inequality of Cove and Hat. IEEE Tansactions on Patten Analysis and Machine Intelligence, 3:75 78, [9] D. P. Dokin and D. G. Kikpatick. Fast detection of poyhedal intesection. Theoetical Compute Science, 27: , [10] D. P. Dokin and D. G. Kikpatick. A linea algoithm fo detemining the sepaation of convex polyheda. Jounal of Algoithms, 6: , [11] C. A. R. Hoae. ACM Algoithm 64: Quicksot. Communications of the ACM, 4(7):321, [12] D. G. Kikpatick and R. Seidel. The ultimate plana convex hull algoithm? SIAM Jounal on Computing, 15(1): , [13] F. P Pepaata and M. I. Shamos. Computational Geomety. Spinge-Velag,

11 [14] M. I. Shamos. Geometic complexity. In Poceedings of the 7th ACM Symposium on the Theoy of Computing (STOC 1975), pages , [15] C. Stone. Consistent nonpaametic egession. Annals of Statistics, 8: , [16] G. T. Toussaint. Poximity gaphs fo instance-ased leaning. Manuscipt, [17] G. T. Toussaint, B. K. Bhattachaya, and R. S. Poulsen. The application of Voonoi diagams to non-paametic decision ules. In Poceedings of Compute Science and Statistics: 16th Symposium of the Inteface, [18] R. Wenge. Randomized quick hull. Algoithmica, 17: ,