Plug-and-Play Dual-Tree Algorithm Runtime Analysis

Transcription

1 Jounal of Machine Leaning Reseach 16 (2015) Submitted 1/15; Published 12/15 Plug-and-Play Dual-Tee Algoithm Runtime Analysis Ryan R. Cutin School of Computational Science and Engineeing Geogia Institute of Technology Atlanta, GA , USA Dongyeol Lee Yahoo Labs Sunnyvale, CA William B. Mach Institute fo Computational Engineeing and Sciences Univesity of Texas, Austin Austin, TX Paikshit Ram Skytee, Inc. Atlanta, GA Edito: Nando de Feitas Abstact Numeous machine leaning algoithms contain paiwise statistical poblems at thei coe that is, tasks that equie computations ove all pais of input points if implemented naively. Often, tee stuctues ae used to solve these poblems efficiently. Dual-tee algoithms can efficiently solve o appoximate many of these poblems. Using cove tees, igoous wostcase untime guaantees have been poven fo some of these algoithms. In this pape, we pesent a poblem-independent untime guaantee fo any dual-tee algoithm using the cove tee, sepaating out the poblem-dependent and the poblem-independent elements. This allows us to just plug in bounds fo the poblem-dependent elements to get untime guaantees fo dual-tee algoithms fo any paiwise statistical poblem without e-deiving the entie poof. We demonstate this plug-and-play pocedue fo neaest-neighbo seach and appoximate kenel density estimation to get impoved untime guaantees. Unde mild assumptions, we also pesent the fist linea untime guaantee fo dual-tee based ange seach. Keywods: dual-tee algoithms, adaptive untime analysis, cove tee, expansion constant, neaest neighbo seach, kenel density estimation, ange seach 1. Dual-tee Algoithms A supising numbe of machine leaning algoithms have computational bottlenecks that can be expessed as paiwise statistical poblems. By this, we mean computational tasks that can be evaluated diectly by iteating ove all pais of input points. Neaest neighbo seach is one such poblem, since fo evey quey point, we can evaluate its distance to evey efeence point and keep the closest one. This naively equies O(N) time to answe a single quey in a efeence set of size N; answeing O(N) queies subsequently equies c 2015 Ryan R. Cutin, Dongyeol Lee, William B. Mach, Paikshit Ram.

2 Cutin, Lee, Mach, and Ram pohibitive O(N 2 ) time. Kenel density estimation is also a paiwise statistical poblem, since we compute a sum ove all efeence points fo each quey point. This again equies O(N 2 ) time to answe O(N) queies if done diectly. The efeence set is typically indexed with spatial data stuctues to acceleate this type of computation (Finkel and Bentley, 1974; Beygelzime et al., 2006); these esult in O(log N) untime pe quey unde favoable conditions. Building upon this intuition, Gay and Mooe (2001) genealized the fast multipole method fom computational physics to obtain dual-tee algoithms. These ae extemely useful when thee ae lage quey sets, not just a few quey points. Instead of building a tee on the efeence set and seaching with each quey point sepaately, Gay and Mooe suggest also building a quey tee and tavesing both the quey and efeence tees simultaneously (a dual-tee tavesal, fom which the class of algoithms takes its name). Dual-tee algoithms can be easily undestood though the ecent famewok of Cutin et al. (2013b): two tees (a quey tee and a efeence tee) ae tavesed by a puning dualtee tavesal. This tavesal visits combinations of nodes fom the tees in some sequence (each combination consisting of a quey node and a efeence node), calling a poblemspecific Scoe() function to detemine if the node combination can be puned. If not, then a poblem-specific BaseCase() function is called fo each combination of points held in the quey node and efeence node. This has significant similaity to the moe common single-tee banch-and-bound algoithms, except that the algoithm must ecuse into child nodes of both the quey tee and efeence tee. Thee exist numeous dual-tee algoithms fo poblems as divese as kenel density estimation (Gay and Mooe, 2003), mean shift (Wang et al., 2007), minimum spanning tee calculation (Mach et al., 2010), n-point coelation function estimation (Mach et al., 2012), max-kenel seach (Cutin et al., 2013c), paticle smoothing (Klaas et al., 2006), vaiational infeence (Amizadeh et al., 2012), ange seach (Gay and Mooe, 2001), and embedding techniques (Van De Maaten, 2014), to name a few. Some of these algoithms ae deived using the cove tee (Beygelzime et al., 2006), a data stuctue with compelling theoetical qualities. When cove tees ae used, dual-tee all-neaest-neighbo seach and appoximate kenel density estimation have O(N) untime guaantees fo O(N) queies (Ram et al., 2009a); minimum spanning tee calculation scales as O(N log N) (Mach et al., 2010). Othe poblems have simila wost-case guaantees (Cutin and Ram, 2014; Mach, 2013). In this wok we combine the genealization of Cutin et al. (2013b) with the theoetical esults of Beygelzime et al. (2006) and othes in ode to develop a wost-case untime bound fo any dual-tee algoithm when the cove tee is used. Section 2 lays out the equied backgound, notation, and intoduces the cove tee and its associated theoetical popeties. Reades familia with the cove tee liteatue and dual-tee algoithms (especially Cutin et al., 2013b) may find that section to be eview. Following that, we intoduce an intuitive measue of cove tee imbalance, an impotant popety fo undestanding the untime of dual-tee algoithms, in Section 3. This measue of imbalance is then used to pove the main esult of the pape in Section 4, which is a wost-case untime bound fo genealized dual-tee algoithms. We apply this esult to thee specific poblems: neaest neighbo seach (Section 5), appoximate kenel density estimation (Section 6), and ange seach / ange count (Section 7), showing linea untime 3270

3 Plug-and-Play Dual-Tee Algoithm Runtime Analysis Symbol N Desciption A tee node Set of child nodes of N i Set of points held in N i Set of descendant nodes of N i Set of points contained in N i and Di n µ i Cente of N i λ i Futhest descendant distance fom µ i C i P i Di n D p i Table 1: Notation fo tees. See Cutin et al. (2013b) fo details. bounds fo each of those algoithms. Each of these bounds is an impovement on the stateof-the-at, and in the case of ange seach, is the fist such bound. Despite the intuition this povides fo the scaling popeties of all dual-tee algoithms 1, it must be kept in mind that these wost-case bounds only apply to dual-tee algoithms that use the cove tee and the standad cove tee tavesal. 2. Peliminaies Fo simplicity, the algoithms consideed in this pape will be pesented in a tee-independent context, as in Cutin et al. (2013b), but the only type of tee we will conside is the cove tee (Beygelzime et al., 2006), and the only type of tavesal we will conside is the cove tee puning dual-tee tavesal, which we will descibe late. As we will be making heavy use of tees, we must establish notation (taken fom Cutin et al., 2013b). The notation we will be using is defined in Table The Cove Tee The cove tee is a leveled hieachical data stuctue oiginally poposed fo the task of neaest neighbo seach by Beygelzime et al. (2006). Each node N i in the cove tee is associated with a single point p i. An adequate desciption is given in thei wok (we have adapted notation slightly): A cove tee T on a dataset S is a leveled tee whee each level is a cove fo the level beneath it. Each level is indexed by an intege scale s i which deceases as the tee is descended. Evey node in the tee is associated with a point in S. Each point in S may be associated with multiple nodes in the tee; howeve, we equie that any point appeas at most once in evey level. Let C si denote the set of points in S associated with the nodes at level s i. The cove tee obeys the following invaiants fo all s i : 1. Dual-tee algoithms using kd-tees and othe types of tees have been obseved to empiically scale linealy fo tasks that take quadatic time without the use of tees; see the empiical esults of Gay and Mooe (2001); Mach et al. (2010); Vladymyov and Caeia-Pepinán (2014); Klaas et al. (2006); Gay and Mooe (2003). 3271

4 Cutin, Lee, Mach, and Ram (Nesting). C si C si 1. This implies that once a point p S appeas in C si then evey lowe level in the tee has a node associated with p. (Coveing tee). Fo evey p i C si 1, thee exists a p j C si such that d(p i, p j ) < 2 s i and the node in level s i associated with p j is a paent of the node in level s i 1 associated with p i. (Sepaation). Fo all distinct p i, p j C si, d(p i, p j ) > 2 s i. As a consequence of this definition, if thee exists a node N i, containing the point p i at some scale s i, then thee will also exist a self-child node N ic containing the point p i at scale s i 1 which is a child of N i. In addition, evey descendant point of the node N i is contained within a ball of adius 2 s i+1 centeed at the point p i ; theefoe, λ i = 2 s i+1 and µ i = p i (Table 1). Note that the cove tee may be intepeted as an infinite-leveled tee, with C containing only the oot point, C = S, and all levels between defined as above. Beygelzime et al. (2006) find this epesentation (which they call the implicit epesentation) easie fo desciption of thei algoithms and some of thei poofs. But clealy, this is not suitable fo implementation; hence, thee is an explicit epesentation in which all nodes that have only a self-child ae coalesced upwads (that is, the node s self-child is emoved, and the childen of that self-child ae taken to be the childen of the node). Figue 1 shows each of the levels of an example cove tee (in the explicit epesentation) on a simple six-point dataset. In this wok, we conside only the explicit epesentation of a cove tee, and do not concen ouselves with the details of tee constuction Expansion Constant The explicit epesentation of a cove tee has a numbe of useful theoetical popeties based on the expansion constant (Kage and Ruhl, 2002); we estate its definition below. Definition 1 Let B S (p, ) be the set of points in S within a closed ball of adius aound some p S with espect to a metic d: B S (p, ) = { S : d(p, ) }. Then, the expansion constant of S with espect to the metic d is the smallest c 2 such that B S (p, 2 ) c B S (p, ) p S, > 0. (1) The expansion constant is used heavily in the cove tee liteatue. It is, in some sense, a notion of instinic dimensionality, most useful in scenaios whee c is independent of the numbe of points in the dataset (Kage and Ruhl, 2002; Beygelzime et al., 2006; Kauthgame and Lee, 2004; Ram et al., 2009a). Note also that if points in S H ae being dawn accoding to a stationay distibution f(x), then c will convege to some finite value c f as S. To see this, define c f as a genealization of the expansion constant fo distibutions. c f 2 is the smallest value such that f(x)dx c f f(x)dx (2) B H (p,2 ) B H (p, ) 2. A batch constuction algoithm is given by Beygelzime et al. (2006), called Constuct. 3272

5 Plug-and-Play Dual-Tee Algoithm Runtime Analysis N a N b p 5 p 2 p 0 p 4 4 p 5 p 2 p 0 p 4 2 N c p 1 p 3 p 1 p 3 2 (a) Root node at scale 1. (b) Nodes at scale 0. N a scale 1 scale 0 1 p 5 p 2 1 N e N b N c scale 1 p 0 p 4 N d N e N d p 1 p 3 N 0 N 4 N 2 N 5 N 1 N 3 scale (c) Nodes at scale 1. (d) Abstact epesentation. Figue 1: Example cove tee on six points in R 2. (a) N a is centeed at p 0 with scale 1. (b) N b and N c ae centeed at p 0 and p 1, espectively, and have scale 0. (c) N d and N e ae centeed at p 0 and p 2, espectively, and have scale 1. The leaves, N 0 though N 6, ae centeed at each of the six points, with scale (and theefoe adius 0). Note that although node N b in subfigue (b) ovelaps node N c, point p 1 only belongs to N c, not N b. Note also that this is only one valid cove tee that could be built on the data; othe configuations ae possible; fo instance, selecting a diffeent oot point gives diffeent valid cove tees. 3273

6 Cutin, Lee, Mach, and Ram fo all p H and > 0 such that B H (p, ) f(x)dx > 0, and with B H(p, ) defined as the closed ball of adius in the space H. As a simple example, take f(x) as a unifom spheical distibution in R d : fo any x 1, f(x) is a constant; fo x > 1, f(x) = 0. It is easy to see that c f in this situation is 2 d, and thus fo some dataset S, c must convege to that value as moe and moe points ae added to S. Closed-fom solutions fo c f fo moe complex distibutions ae less easy to deive; howeve, empiical speedup esults fom Beygelzime et al. (2006) suggest the existence of datasets whee c is not stongly dependent on d. Fo instance, the covtype dataset has 54 dimensions but the expansion constant is much smalle than othe, lowe-dimensional datasets. Thee ae some othe impotant obsevations about the behavio of c. Adding a single point to S may incease c abitaily: conside a set S distibuted entiely on the suface of a unit hypesphee. If one adds a single point at the oigin, poducing the set S, then c explodes to S wheeas befoe it may have been much smalle than S. Adding a single point may also decease c significantly. Suppose one adds a point abitaily close to the oigin to S ; now, the expansion constant will be S /2. Both of these situations ae degeneate cases not commonly encounteed in eal-wold behavio; we discuss them in ode to point out that although we can bound the behavio of c as S fo S fom a stationay distibution, we ae not able to easily say much about its convegence behavio. The expansion constant can be used to show a few useful bounds on vaious popeties of the cove tee; we estate these esults below, given some cove tee built on a dataset S with expansion constant c and S = N: Width bound: no cove tee node has moe than c 4 childen (Lemma 4.1, Beygelzime et al., 2006). Depth bound: the maximum depth of any node is O(c 2 log N) (Lemma 4.3, Beygelzime et al., 2006). Space bound: a cove tee has O(N) nodes (Theoem 1, Beygelzime et al., 2006). Lastly, we intoduce a convenience lemma of ou own which is a genealization of the packing aguments used by Beygelzime et al. (2006). This is a moe flexible vesion of thei agument. Lemma 1 Conside a dataset S with expansion constant c and a subset C S such that evey two distinct points in C ae sepaated by at least δ. Then, fo any point p (which may o may not be in S), and any adius ρδ > 0: B S (p, ρδ) C c 2+ log 2 ρ. (3) Poof The poof is based on the packing agument fom Lemma 4.1 in Beygelzime et al. (2006). Conside two cases: fist, let d(p, p i ) > ρδ fo any p i S. In this case, B S (p, ρδ) = and the lemma holds tivially. Othewise, let p i S be a point such that d(p, p i ) ρδ. Obseve that B S (p, ρδ) B S (p i, 2ρδ). Also, B S (p i, 2ρδ) c 2+ log 2 ρ B S (p i, δ/2) by the 3274

7 Plug-and-Play Dual-Tee Algoithm Runtime Analysis definition of the expansion constant. Because each point in C is sepaated by δ, the numbe of points in B S (p, ρδ) C is bounded by the numbe of disjoint balls of adius δ/2 that can be packed into B S (p, ρδ). In the wost case, this packing is pefect, and B S (p, ρδ) B S(p i, 2ρδ) B S (p i, δ/2) c2+ log 2 ρ. (4) 3. Tee Imbalance It is well-known that imbalance in tees leads to degadation in pefomance; fo instance, a kd-tee node with evey descendant in its left child except one is effectively useless. A kd-tee full of nodes like this will pefom abysmally fo neaest neighbo seach, and it is not had to geneate a pathological dataset that will cause a kd-tee of this sot. This sot of imbalance applies to all types of tees, not just kd-tees. In ou situation, we ae inteested in a bette undestanding of this imbalance fo cove tees, and thus endeavo to intoduce a moe fomal measue of imbalance which is coelated with tee pefomance. Numeous measues of tee imbalance have aleady been established; one example is that poposed by Colless (1982), and anothe is Sackin s index (Sackin, 1972), but we aim to captue a diffeent measue of imbalance that uses the leveled stuctue of the cove tee. We aleady know each node in a cove tee is indexed with an intege level (o scale). In the explicit epesentation of the cove tee, each non-leaf node has childen at a lowe level. But these childen need not be stictly one level lowe; see Figue 2. In Figue 2a, each cove tee node has childen that ae stictly one level lowe; we will efe to this as a pefectly balanced cove tee. Figue 2b, on the othe hand, contains the node N m which has two childen with scale two less than s m. We will efe to this as an imbalanced cove tee. Note that in ou definition, the balance of a cove tee has nothing to do with diffeing numbe of descendants in each child banch but instead only missing levels. An imbalanced cove tee can happen in pactice, and in the wost cases, the imbalance may be fa wose than the simple gaphs of Figue 2. Conside a dataset with a single N a s a N m s m s a 1 s m 1 N b N c N d N n N e N f N g N h N i N j s a 2 N k N p N q N N s s m 2 N t (a) Balanced cove tee. (b) Imbalanced cove tee. Figue 2: Balanced and imbalanced cove tees. 3275

8 Cutin, Lee, Mach, and Ram outlie Figue 3: Single-outlie cove tee. Figue 4: A multiple-outlie cove tee. outlie which is vey fa away fom all of the othe points 3. Figue 3 shows what happens in this situation: the oot node has two childen; one of these childen has only the outlie as a descendant, and the othe child has the est of the points in the dataset as a descendant. In fact, it is easy to find datasets with a handful of outlies that give ise to a chain-like stuctue at the top of the tee: see Figue 4 fo an illustation 4. A tee that has this chain-like stuctue all the way down, which is simila to the kd-tee example at the beginning of this section, is going to pefom hoendously; motivated by this obsevation, we define a measue of tee imbalance. Definition 2 The cove node imbalance I n (N i ) fo a cove tee node N i with scale s i in the cove tee T is defined as the cumulative numbe of missing levels between the node and its paent N p (which has scale s p ). If the node is a leaf (that is, s i = ), then the numbe of missing levels is defined as the diffeence between s p and s min 1 whee s min is the smallest scale of a non-leaf node in T. If N i is the oot of the tee, then the cove node imbalance is 0. Explicitly witten, this calculation is s p s i 1 if N i is not a leaf and not the oot node I n (N i ) = max(s p s min 1, 0) if N i is a leaf 0 if N i is the oot node. 3. Note also that fo an outlie sufficiently fa away, the expansion constant is N 1, so we should expect poo pefomance with the cove tee anyway. 4. As a side note, this behavio is not limited to cove tees, and can happen to mean-split kd-tees too, especially in highe dimensions. In addition, fo this scenaio to aise with cove tees, it must be tue that c O(N). (5) 3276

9 Plug-and-Play Dual-Tee Algoithm Runtime Analysis Imbalance Dataset d N = 5k N = 50k N = 500k lcdm sdss powe susy andu higgs covetype mnist Table 2: Empiically calculated tee imbalances, nomalized by N. This simple definition of cove node imbalance is easy to calculate, and using it, we can genealize to a measue of imbalance fo the full tee. Definition 3 The cove tee imbalance I t (T ) fo a cove tee T is defined as the cumulative numbe of missing levels in the tee. This can be expessed as a function of cove node imbalances easily: I t (T ) = N i T I n (N i ). (6) A pefectly balanced cove tee T b with no missing levels has imbalance I t (T b ) = 0 (fo instance, Figue 2a). A wost-case cove tee T w which is entiely a chain-like stuctue with maximum scale s max and minimum scale s min will have imbalance I t (T w ) N(s max s min ). Because of this chain-like stuctue, each level has only one node and thus thee ae at least N levels; o, s max s min N, meaning that in the wost case the imbalance is quadatic in N. 5 Howeve, fo most eal-wold datasets with the cove tee implementation in mlpack (Cutin et al., 2013a) and the efeence implementation (Beygelzime et al., 2006), the tee imbalance is nea-linea with the numbe of points. We have constucted cove tees on N unifomly subsampled points fom a vaiety of datasets and calculated the imbalance; see Table 2 fo the esults. Ten tials wee pefomed fo each dataset and each N, and the mean imbalance is given. These esults ae nomalized with espect to N, fo which the values of 5000, 50000, and wee chosen. The powe, susy, higgs, and covetype datasets ae found in the UCI Machine Leaning Repositoy (Bache and Lichman, 2013), the mnist dataset is fom LeCun et al. (2000), the lcdm and sdss datasets ae Sloan Digital Sky Suvey data (Adelman-McCathy et al., 2008), and the andu dataset is andomlygeneated unifomly-distibuted data in 10 dimensions. The imbalances on each of these datasets tend to be nea-linea. Cuently, no cove tee constuction algoithm specifically aims to minimize imbalance. 5. Note that in this situation, c N also. 3277

10 Cutin, Lee, Mach, and Ram Algoithm 1 The standad puning dual-tee tavesal fo cove tees. 1: Input: quey node N q, set of efeence nodes R 2: Output: none max N R s 4: if (s q < s max ) then 5: {Pefom a efeence ecusion.} 6: fo each N R do 7: BaseCase(p q, p ) 8: end fo 9: R {N R : s = s max } 10: R 1 {C (N ) : N R } (R \ R ) 11: R 1 {N R 1 : Scoe(N q,n ) } 3: s max 12: ecuse with N q and R 1 13: else 14: {Pefom a quey ecusion.} 15: fo each N qc C (N q ) do 16: R {N R : Scoe(N qc,n ) } 17: ecuse with N qc and R 18: end fo 19: end if 4. Geneal Runtime Bound Pehaps moe inteesting than measues of tee imbalance is the way cove tees ae actually used in dual-tee algoithms. Although cove tees wee oiginally intended fo neaest neighbo seach (See Algoithm Find-All-Neaest, Beygelzime et al., 2006), they can be adapted to a wide vaiety of poblems: minimum spanning tee calculation (Mach et al., 2010), appoximate neaest neighbo seach (Ram et al., 2009b), Gaussian pocesses posteio calculation (Mooe and Russell, 2014), and max-kenel seach (Cutin and Ram, 2014) ae some examples. Futhe, though the tee-independent dual-tee algoithm abstaction of Cutin et al. (2013b), othe existing dual-tee algoithms can easily be adapted fo use with cove tees. In the famewok of tee-independent dual-tee algoithms, all that is necessay to descibe a dual-tee algoithm is a point-to-point base case function (BaseCase()) and a node-to-node puning ule (Scoe()). These functions, which ae often vey staightfowad, ae then paied with a type of tee and a puning dual-tee tavesal to poduce a woking algoithm. In late sections, we will conside specific examples. When using cove tees, the typical puning dual-tee tavesal is an adapted fom of the oiginal neaest neighbo seach algoithm (see Find-All-Neaest, Beygelzime et al., 2006); this tavesal is implemented in both the cove tee efeence implementation and in the moe flexible mlpack libay (Cutin et al., 2013a). The poblem-independent tavesal is given in Algoithm 1 and was oiginally pesented by Cutin and Ram (2014). Initially, it is called with the oot of the quey tee and a efeence set R containing only the oot of the efeence tee. 3278

11 Plug-and-Play Dual-Tee Algoithm Runtime Analysis This dual-tee ecusion is a depth-fist ecusion in the quey tee and a beadth-fist ecusion in the efeence tee; to this end, the ecusion maintains one quey node N q and a efeence set R. The set R may contain efeence nodes with many diffeent scales; the maximum scale in the efeence set is s max (line 3). Each single ecusion will descend eithe the quey tee o the efeence tee, not both; the conditional in line 4, which detemines whethe the quey o efeence tee will be ecused, is aimed at keeping the elative scales of quey nodes and efeence nodes close. Keeping the quey and efeence scales close is both beneficial fo the late theoy and intuitively easonable: ecusing too quickly in the eithe the quey o efeence node will unnecessaily duplicate wok. Suppose we ecuse many levels down the quey tee befoe ecusing down the efeence tee, giving us a set of quey nodes we ae consideing. Fo each of these quey nodes, we will then need to descend the efeence tee. Because these quey nodes ae close togethe (with espect to the efeence nodes we ae consideing, which ae of lage scale and thus futhe apat), the puning decisions at each level of ecusion ae likely to be the same fo each quey node. Theefoe, ecusing too fa in the quey tee may cause a lage amount of duplicated wok. The symmetic agument applies fo ecusing too fa in the efeence tee befoe ecusing in the quey tee. This justifies the appoach of keeping the quey and efeence scales appoximately equal. A quey ecusion (lines 13 18) is staightfowad: fo each child N qc of N q, the node combinations (N qc, N ) ae scoed fo each N in the efeence set R. If possible, these combinations ae puned to fom the set R (line 17) by checking the output of the Scoe() function, and then the algoithm ecuses with N qc and R. A efeence ecusion (lines 4 12) is simila to a quey ecusion, but the puning stategy is significantly moe complicated. Given R, we calculate R, which is the set of nodes in R that have scale s max. We expand each node in R to constuct R 1 : this is the set of childen of all nodes in R. This set is then combined with R \ R (that is, the set of efeences nodes not at scale s max ) to poduce R 1. Each node in R 1 is then scoed and puned if possible, esulting in the puned efeence set R 1. The algoithm then ecuses with N q and R 1. The efeence ecusion only ecuses into the top-level subset of the efeence nodes in ode to peseve the sepaation invaiant. It is easy to show that evey pai of points held in nodes in R is sepaated by at least 2 smax : Lemma 2 Fo all distinct nodes N i, N j R (in the context of Algoithm 1) which contain points p i and p j, espectively, d(p i, p j ) > 2 smax, with s max defined as in line 3. Poof This poof is by induction. If R = 1, such as duing the fist efeence ecusion, the esult obviously holds. Now conside any efeence set R and assume the statement of the lemma holds fo this set R, and define s max as the maximum scale of any node in R. Constuct the set R 1 as in line 10 of Algoithm 1; if R 1 1, then R 1 satisfies the desied popety. Othewise, take any N i, N j in R 1, with points p i and p j, espectively, and scales s i and s j, espectively. Clealy, if s i = s j = s max 1, then by the sepaation invaiant d(p i, p j ) > 2 smax 1. Now suppose that s i < s max 1. This implies that thee exists some implicit cove tee node with point p i and scale s max 1 (as well as an implicit child of this node p i with scale 3279

12 Cutin, Lee, Mach, and Ram 2 and so foth until one of these implicit nodes has child p i with scale s i ). Because the sepaation invaiant applies to both implicit and explicit epesentations of the tee, we conclude that d(p i, p j ) > 2 smax 1. The same agument may be made fo the case whee s j < s max 1, with the same conclusion. We may theefoe conclude that each point of each node in R 1 is sepaated by 2 smax 1. Note that R 1 R 1 and that R \ R 1 R in ode to see that this condition holds fo all nodes in R 1. Because we have shown that the condition holds fo the initial efeence set and fo any efeence set poduced by a efeence ecusion (which will be R at some othe level of ecusion), we have shown that the statement of the lemma is tue. Note that in this poof, we have consideed the child efeence set R 1, not the oiginal s max efeence set R, and shown that with espect to s max as defined by R (not R 1 ), all nodes ae sepaated by 2 smax 1. Then, in the fame of the next ecusion whee R R 1, the lemma will hold, as s max will then be the maximum scale pesent in R. This obsevation means that the set of points P held by all nodes in R is always a subset of C s max. This fact will be useful in ou late untime poofs. Next, we develop notions with which to undestand the behavio of the cove tee dualtee tavesal when the datasets ae of significantly diffeent scale distibutions. If the datasets ae simila in scale distibution (that is, inte-point distances tend to follow the same distibution), then the ecusion will altenate between quey ecusions and efeence ecusions. But if the quey set contains points which ae, in geneal, much fathe apat than the efeence set, then the ecusion will stat with many quey ecusions befoe eaching a efeence ecusion. The convese case also holds. We ae inteested in fomalizing this notion of scale distibution; theefoe, define the following dataset-dependent constants fo the quey set S q and the efeence set S : η q : the lagest paiwise distance in S q δ q : the smallest nonzeo paiwise distance in S q η : the lagest paiwise distance in S δ : the smallest nonzeo paiwise distance in S These constants ae diectly elated to the aspect atio of the datasets; indeed, η q /δ q is exactly the aspect atio of S q. Futhe, let us define and bound the top and bottom levels of each tee: The top scale s T q of the quey tee T q is such that as log 2 (η q ) 1 s T q log 2 (η q ). The minimum scale of the quey tee T q is defined as s min q = log 2 (δ q ). The top scale s T of the efeence tee T is such that as log 2 (η ) 1 s T log 2 (η ). The minimum scale of the efeence tee T is defined as s min = log 2 (δ ). 3280

13 Plug-and-Play Dual-Tee Algoithm Runtime Analysis Note that the minimum scale is not the minimum scale of any cove tee node (that would be ), but the minimum scale of any non-leaf node in the tee. Suppose that ou datasets ae of a simila scale distibution: s T q = s T, and s min q = s min. In this setting we will have altenating quey and efeence ecusions. But if this is not the case, then we have exta efeence ecusions befoe the fist quey ecusion o afte the last quey ecusion (situations whee both these cases happen ae possible). Motivated by this obsevation, let us quantify these exta efeence ecusions: Lemma 3 Fo a dual-tee algoithm with S q S O(N) using cove tees and the tavesal given in Algoithm 1, the numbe of exta efeence ecusions that happen befoe the fist quey ecusion is bounded by min (O(N), log 2 (η /η q ) 1). (7) Poof The fist quey ecusion happens once s q s max. The numbe of efeence ecusions befoe the fist quey ecusion is then bounded as the numbe of levels in the efeence tee between s T and s T q that have at least one explicit node. Because thee ae O(N) nodes in the efeence tee, the numbe of levels cannot be geate than O(N) and thus the esult holds. The second bound holds by applying the definitions of s T and s T q to the expession s T s T q 1: s T s T q 1 log 2 (η ) ( log 2 (η q ) 1) 1 (8) log 2 (η ) + 1 log 2 (η q ) (9) which gives the statement of the lemma afte applying logaithmic identities. Note that the O(N) bound may be somewhat loose, but it suffices fo ou late puposes. Now let us conside the othe case: Lemma 4 Fo a dual-tee algoithm with S q S O(N) using cove tees and the tavesal given in Algoithm 1, the numbe of exta efeence ecusions that happen afte the last quey ecusion is bounded by max ( min ( O(N log 2 (δ q /δ )), O(N 2 ) ), 0 ). (10) Fo convenience, we define a tem that encapsulates this bound. Definition 4 Define θ as a bound on the numbe of exta efeence ecusions that happen afte the last quey ecusion. Then, θ = max { min ( O(N log 2 (δ q /δ )), O(N 2 ) ), 0 }. (11) 3281

14 Cutin, Lee, Mach, and Ram Poof Ou goal hee is to count the numbe of efeence ecusions afte the final quey ecusion at level s min q ; the fist of these efeence ecusions is at scale s max = s min q. Because quey nodes ae not puned in this tavesal, each efeence ecusion we ae counting will be duplicated ove the whole set of O(N) quey nodes. The fist pat of the bound follows by obseving that s min q s min log 2 (δ q ) log 2 (δ ) 1 log 2 (δ q /δ ). The second pat follows by simply obseving that thee ae O(N) efeence nodes. These two pevious lemmas allow us a bette undestanding of what happens as the efeence set and quey set become diffeent. Lemma 3 shows that the numbe of exta ecusions caused by a efeence set with lage paiwise distances than the quey set (η lage than η q ) is modest; on the othe hand, Lemma 4 shows that fo each exta level in the efeence tee below s min q, O(N) exta ecusions ae equied. Using these lemmas and this intuition, we will pove geneal untime bounds fo the cove tee tavesal. Theoem 1 Given a efeence set S of size O(N) with an expansion constant c and a set of queies S q of size O(N), a standad cove tee based dual-tee algoithm (Algoithm 1) takes O ( c 4 R χψ(n + I t (T q ) + θ) ) (12) time, whee R is the maximum size of the efeence set R (line 1) duing the dual-tee ecusion, χ is the maximum possible untime of BaseCase(), ψ is the maximum possible untime of Scoe(), and θ is defined as in Lemma 4. Poof Fist, split the algoithm into two pats: efeence ecusions (lines 4 12) and quey ecusions (lines 13 18). The untime of the algoithm is the untime of a efeence ecusion times the total numbe of efeence ecusions plus the total untime of all quey ecusions. Conside a efeence ecusion (lines 4 12). Define R to be the lagest set R fo any and any quey node N q duing the couse of the algoithm; then, it is tue that R R. The wok done in the base case loop fom lines 6 8 is thus O(χ R ) O(χ R ). Then, lines 10 and 11 take O(c 4 ψ R ) O(c 4 ψ R ) time, because each efeence node has up to c 4 childen. So, one full efeence ecusion takes O(c 4 ψχ R ) time. scale s max Now, note that thee ae O(N) nodes in T q. Thus, line 17 is visited O(N) times. The amount of wok in line 16, like in the efeence ecusion, is bounded as O(c 4 ψ R ). Theefoe, the total untime of all quey ecusions is O(c 4 ψ R N). Lastly, we must bound the total numbe of efeence ecusions. Refeence ecusions happen in thee cases: (1) s max is geate than the scale of the oot of the quey tee (no quey ecusions have happened yet); (2) s max is less than o equal to the scale of the oot of the quey tee, but is geate than the minimum scale of the quey tee that is not ; (3) s max is less than the minimum scale of the quey tee that is not. Fist, conside case (1). Lemma 3 shows that the numbe of efeence ecusions of this type is bounded by O(N). Although thee is also a bound that depends on the sizes of the datasets, we only aim to show a linea untime bound, so the O(N) bound is sufficient hee. Next, conside case (2). In this situation, each quey ecusion implies at least one efeence ecusion befoe anothe quey ecusion. Fo some quey node N q, the exact numbe of efeence ecusions befoe the childen of N q ae ecused into is bounded above 3282

15 Plug-and-Play Dual-Tee Algoithm Runtime Analysis by I n (N q ) + 1: if N q has imbalance 0, then it is exactly one level below its paent, and thus thee is only one efeence ecusion. On the othe hand, if N q is many levels below its paent, then it is possible that a efeence ecusion may occu fo each level in between; this is a maximum of I n (N q ) + 1. Because each quey node in T q is ecused into once, the total numbe of efeence ecusions befoe each quey ecusion is N q T q I n (N q ) + 1 = I t (T q ) + O(N) (13) since thee ae O(N) nodes in the quey tee. Lastly, fo case (3), we may efe to Lemma 4, giving a bound of θ efeence ecusions in this case. We may now combine these esults fo the untime of a quey ecusions with the total numbe of efeence ecusions in ode to give the esult of the theoem: O ( c 4 R ψχ (N + I t (T q ) + θ) ) + O ( c 4 R ψn ) O ( c 4 R ψχ (N + I t (T q ) + θ) ). (14) When we conside the monochomatic case (whee S q = S ), the esults tivially simplify. Coollay 1 Given the situation of Theoem 1 but with S q = S = S so that c q = c = c and T q = T = T, a dual-tee algoithm using the standad cove tee tavesal (Algoithm 1) takes O ( c 4 R χψ (N + I t (T )) ) (15) time, whee R is the maximum size of the efeence set R (line 1) duing the dualtee ecusion, χ is the maximum possible untime of BaseCase(), and ψ is the maximum possible untime of Scoe(). An intuitive undestanding of these bounds is best achieved by fist consideing the monochomatic case (this case aises, fo instance, in all-neaest-neighbo seach). The linea dependence on N aises fom the fact that all quey nodes must be visited. The dependence on the efeence tee, howeve, is encapsulated by the tem c 4 R, with R being the maximum size of the efeence set R; this value must be deived fo each specific poblem. The poo pefomance of tees on datasets with lage c (o, in the wost case, c N) is then captued in both of those tems. These datasets fo which tees pefom pooly may also have a high cove tee imbalance I t (T ); the linea dependence of untime on imbalance is thus sensible fo datasets whee tees pefom well. The bichomatic case (S q S ) is a slightly moe complex esult which deseves a bit moe attention. The intuition fo all tems except θ emain vitually the same. The tem θ captues the effect of quey and efeence datasets with diffeent widths, and has one unfotunate cone case: when δ q > η, then the quey tee must be entiely descended befoe any efeence ecusion. This esults in a bound of the fom O(N log(η /δ )), 3283

16 Cutin, Lee, Mach, and Ram o O(N 2 ) (see Lemma 4). This is because the efeence tee must be descended sepaately fo each quey point. The quantity R bounds the amount of wok that needs to be done fo each ecusion. In the wost case, R can be N. Howeve, dual-tee algoithms ely on banch-and-bound techniques to pune away wok (lines 11 and 16 in Algoithm 1). A small value of R will imply that the algoithm is extemely successful in puning away wok. An (uppe) bound on R (and the algoithm s success in puning wok) will depend on the poblem and the data. As we will show, bounding R is often possible. Fo many dual-tee algoithms, χ ψ O(1); often, cached sufficient statistics (Mooe, 2000) can enable O(1) untime implementations of BaseCase() and Scoe(). These esults hold fo any dual-tee algoithm egadless of the poblem. Hence, the untime of any dual-tee algoithm can be bounded no moe tightly than O(N) with ou bound, which matches the intuition that answeing O(N) queies will take at least O(N) time. Fo a paticula poblem and data, if c, R, χ, and ψ ae bounded by constants independent of N and θ is no moe than linea in N (fo lage enough N), then the dual-tee algoithm fo that poblem has a untime linea in N. Ou theoetical esult sepaates out the poblem-dependent and the poblem-independent elements of the untime bound, which allows us to simply plug in the poblem-dependent bounds in ode to get untime bounds fo any dual-tee algoithm without equiing an analysis fom scatch. Ou esults ae simila to that of Ram et al. (2009a), but those esults depend on a quantity called the constant of bichomaticity, denoted κ, which has unclea elation to cove tee imbalance. The dependence on κ is given as c 4κ q, which is not a good bound, especially because κ may be much geate than 1 in the bichomatic case (whee S q S ). The moe ecent esults of Cutin and Ram (2014) ae moe elated to these esults, but they depend on the invese constant of bichomaticity ν which suffes fom the same poblem as κ. Although the dependence on ν is linea (that is, O(νN)), bounding ν is difficult and it is not tue that ν = 1 in the monochomatic case. The quantity ν coesponds to the maximum numbe of efeence ecusions between a single quey ecusion, and κ coesponds to the maximum numbe of quey ecusions between a single efeence ecusion. The espective poofs that use these constants then apply them as a wost-case measue fo the whole algoithm: when using κ, Ram et al. (2009a) assume that evey efeence ecusion may be followed by κ quey ecusions; similaly, Cutin and Ram (2014) assume that evey quey ecusion may be followed by ν efeence ecusions. Hee, we have simply used I t (T q ) and θ as an exact summation of the total exta efeence ecusions, which gives us a much tighte bound than ν o κ on the unning time of the whole algoithm. Futhe, both ν and κ ae difficult to empiically calculate and equie an entie un of the dual-tee algoithm. On the othe hand, bounding I t (T q ) (and θ) can be done in one pass of the tee (assuming the tee is aleady built). Thus, not only is ou bound tighte when the cove tee imbalance is sublinea in N, it moe closely eflects the actual behavio of dual-tee algoithms, and the constants which it depends upon ae staightfowad to calculate. In the following sections, we will apply ou esults to specific poblems and show the utility of ou bound in simplifying untime poofs fo dual-tee algoithms. 3284

17 Plug-and-Play Dual-Tee Algoithm Runtime Analysis Algoithm 2 Neaest neighbo seach BaseCase() Input: quey point p q, efeence point p, list of candidate neighbos N and distances D Output: distance d between p q and p if d(p q, p ) < D[p q ] and BaseCase(p q, p ) not yet called then D[p q ] d(p q, p ), and N[p q ] p end if etun d(p q, p ) Algoithm 3 Neaest neighbo seach Scoe() Input: quey node N q, efeence node N Output: a scoe fo the node combination (N q, N ), o if the combination should be puned if d min (N q, N ) < B(N q ) then etun d min (N q, N ) end if etun 5. Neaest Neighbo Seach The standad task of neaest neighbo seach can be simply descibed: given a quey set S q and a efeence set S, fo each quey point p q S q, find the neaest neighbo p in the efeence set S. The task is well-studied and well-known, and thee exist numeous appoaches fo both exact and appoximate neaest neighbo seach, including the cove tee neaest neighbo seach algoithm due to Beygelzime et al. (2006). We will conside that algoithm, but in a tee-independent sense as given by Cutin et al. (2013b); this means that to descibe the algoithm, we equie only a BaseCase() and Scoe() function; these ae given in Algoithms 2 and 3, espectively. The point-to-point BaseCase() function compaes a quey point p q and a efeence point p, updating the list of candidate neighbos fo p q if necessay. The node-to-node Scoe() function detemines if the entie subtee of nodes unde the efeence node N can impove the candidate neighbos fo all descendant points of the quey node N q ; if not, the node combination is puned. The Scoe() function depends on the function d min (, ), which epesents the minimum possible distance between any two descendants of two nodes. Its definition fo cove tee nodes is d min (N q, N ) = d(p q, p ) 2 sq+1 2 s+1. (16) Given a type of tee and tavesal, these two functions stoe the cuent neaest neighbo candidates in the aay N and thei distances in the aay D. (See Cutin et al., 2013b, fo a moe complete discussion of how this algoithm woks and a poof of coectness.) The Scoe() function depends on a bound function B(N q ) which epesents the maximum distance that could possibly impove a neaest neighbo candidate fo any descendant point 3285

18 Cutin, Lee, Mach, and Ram of the quey node N q. The standad bound function B(N q ) used fo cove tees is adapted fom Beygelzime et al. (2006): B(N q ) := D[p q ] + 2 sq+1 (17) In this fomulation, the quey node N q holds the the quey point p q, the quantity D[p q ] is the cuent neaest neighbo candidate distance fo the quey point p q, and 2 sq+1 coesponds to the futhest descendant distance of N q. Fo notational convenience in the following poof, take c q = max((max pq Sq c ), c ), whee c is the expansion constant of the set S {p q }. Theoem 2 Using cove tees, the standad cove tee puning dual-tee tavesal, and the neaest neighbo seach BaseCase() and Scoe() as given in Algoithms 2 and 3, espectively, and also given a efeence set S of size O(N) with expansion constant c, and a quey set S q of size O(N), the unning time of the algoithm is bounded by O(c 4 c 5 q(n +I t (T q )+θ)) with I t (T q ) and θ defined as in Definition 3 and Lemma 4, espectively. Poof The unning time of BaseCase() and Scoe() ae clealy O(1). Due to Theoem 1, we theefoe know that the untime of the algoithm is bounded by O(c 4 R (N +I t (T q )+θ)). Thus, the only thing that emains is to bound the maximum size of the efeence set, R. Assume that when R is encounteed, the maximum efeence scale is s max and the quey node is N q. Evey node N R satisfies the popety enfoced in line 11 that d min (N q, N ) B(N q ). Using the definition of d min (, ) and B( ), we expand the equation. Note that p q is the point held in N q and p is the point held in N. Also, take ˆp to be the cuent neaest neighbo candidate fo p q ; that is, D[p q ] = d(p q, ˆp ) and N[p q ] = ˆp. Then, d min (N q, N ) B(N q ) (18) d(p q, p ) d(p q, ˆp ) + 2 sq s sq+1 (19) whee the last step follows because s q + 1 s max d(p q, ˆp ) + 2(2 smax +1 ) (20) and s s max. Define the set of points P as the points held in each node in R (that is, P = {p P(N ) : N R }). Then, we can wite P B S (p q, d(p q, ˆp ) + 2(2 smax +1 )). (21) Suppose that the tue neaest neighbo is p and d(p q, p ) > 2 smax +1. Then, p must be held as a descendant point of some node in R which holds some point p. Using the tiangle inequality, d(p q, ˆp ) d(p q, p ) d(p q, p ) + d( p, p ) d(p q, p ) + 2 smax +1. (22) This gives that P B S {pq}(p q, d(p q, p ) + 3(2 smax +1 )). The pevious step is necessay: to apply the definition of the expansion constant, the ball must be centeed at a point in the set; now, the cente (p q ) is pat of the set. 3286

19 Plug-and-Play Dual-Tee Algoithm Runtime Analysis B S {p q}(p q, d(p q, p ) + 3(2 smax +1 )) B S {p q}(p q, 4d(p q, p )) (23) c 3 q B S {p q}(p q, d(p q, p )/2) (24) which follows because the expansion constant of the set S {p q } is bounded above by c q. Next, we know that p is the closest point to p q in S {p q }; thus, thee cannot exist a point p p q S {p q } such that p B Sq (p q, d(p q, p )/2) because that would imply that d(p q, p ) < d(p q, p ), which is a contadiction. Thus, the only point in the ball is p q, and we have that B S {pq}(p q, d(p q, p )/2) = 1, giving the esult that R c 3 q in this case. The othe case is when d(p q, p ) 2 smax, and theefoe that P C s max +1, which means that d(p q, ˆp ) 2 smax +2. Note P B S (p q, d(p q, p ) + 3(2 smax +1 )) C s max (25) B S (p q, 4(2 smax +1 )) C s max. (26) Evey point in C s max is sepaated by at least 2 smax. Using Lemma 1 with δ = 2 smax ρ = 8 yields that P c 5. This gives the esult, because c 5 c 5 q. and In the monochomatic case whee S q = S 6, the bound is O(c 9 (N + I t (T )) because c = c = c q and θ = 0. Fo well-behaved tees whee I t (T q ) is linea o sublinea in N, this epesents the cuent tightest wost-case untime bound fo neaest neighbo seach. 6. Appoximate Kenel Density Estimation Ram et al. (2009a) pesent a cleve technique fo bounding the unning time of appoximate kenel density estimation based on the popeties of the kenel, when the kenel is shiftinvaiant and satisfies a few assumptions. We will estate these assumptions and povide an adapted poof using Theoem 1, which gives a tighte bound. Appoximate kenel density estimation is a common application of dual-tee algoithms (Gay and Mooe, 2003, 2001). Given a quey set S q, a efeence set S of size N, and a kenel function K(, ), the tue kenel density estimate fo a quey point p q is given as f (p q ) = p S K(p q, p ). (27) In the case of an infinite-tailed kenel K(, ), the exact computation cannot be acceleated; thus, attention has tuned towads tactable appoximation schemes. Two simple schemes fo the appoximation of f (p q ) ae well-known: absolute value appoximation and elative value appoximation. Absolute value appoximation equies that each density estimate f(p q ) is within ɛ of the tue estimate f (p q ): f(p q ) f (p q ) < ɛ p q S q. (28) 6. In the monochomatic case, we do not take a point as its own neaest neighbo, so slight modification of BaseCase() is necessay. The untime bound esult emains unchanged. 3287

20 Cutin, Lee, Mach, and Ram Relative value appoximation is a moe flexible appoximation scheme; given some paamete ɛ, the equiement is that each density estimate is within a elative toleance of f (p q ) : f(p q ) f (p q ) f (p q ) < ɛ p q S q. (29) Kenel density estimation is elated to the well-studied poblem of kenel summation, which can also be solved with dual-tee algoithms (Lee and Gay, 2006, 2009). In both of those poblems, egadless of the appoximation scheme, simple geometic obsevations can be made to acceleate computation: when K(, ) is shift-invaiant, faaway points have vey small kenel evaluations. Thus, tees can be built on S q and S, and node combinations can be puned when the nodes ae fa apat while still obeying the eo bounds. In the following two subsections, we will sepaately conside both the absolute value appoximation scheme and the elative value appoximation scheme, unde the assumption of a shift-invaiant kenel K(p q, p ) = K( p q p ) which is monotonically deceasing and non-negative. In addition, we assume that thee exists some bandwidth h such that K(d) must be concave fo d [0, h] and convex fo d [h, ). This assumption implies that the magnitude of the deivative K (d) is maximized at d = h. These ae not estictive assumptions; most standad kenels fall into this class, including the Gaussian, exponential, and Epanechnikov kenels. 6.1 Absolute Value Appoximation A tee-independent algoithm fo solving appoximate kenel density estimation with absolute value appoximation unde the pevious assumptions on the kenel is given as a BaseCase() function in Algoithm 4 and a Scoe() function in Algoithm 5 (a coectness poof can be found in Cutin et al., 2013b). The list f p holds patial kenel density estimates fo each quey point, and the list f n holds patial kenel density estimates fo each quey node. At the beginning of the dual-tee tavesal, the lists f p and f n, which ae both of size O(N), ae each initialized to 0. As the tavesal poceeds, node combinations ae puned if the diffeence between the maximum kenel value K(d min (N q, N )) and the minimum kenel value K(d max (N q, N )) is sufficiently small (line 3). If the node combination can be puned, then the patial node estimate is updated (line 4). When node combinations cannot be puned, BaseCase() may be called, which simply updates the patial point estimate with the exact kenel evaluation (line 3). Afte the dual-tee tavesal, the actual kenel density estimates f must be extacted. This can be done by tavesing the quey tee and calculating f(p q ) = f p (p q )+ N i T f n(n i ), whee T is the set of nodes in T q that have p q as a descendant. Each quey node needs to be visited only once to pefom this calculation; it may theefoe be accomplished in O(N) time. Note that this vesion is fa simple than othe dual-tee algoithms that have been poposed fo appoximate kenel density estimation (see, fo instance, Gay and Mooe, 2003); howeve, this vesion is sufficient fo ou untime analysis. Real-wold implementations, such as the one found in mlpack (Cutin et al., 2013a), tend to be fa moe complex. 3288