Fast Resampling Weighted v-statistics

Transcription

1 Fast Resampling Weighte v-statistics Chunxiao Zhou Mar O. Hatfiel Clinical Research Center National Institutes of Health Bethesa, MD Jiseong Par Dept of Math George Mason Univ Fairfax, VA Yun Fu Dept of ECE Northeastern Univ Boston, MA 025 Abstract In this paper, a novel an computationally fast algorithm for computing weighte v-statistics in resampling both univariate an multivariate ata is propose. To avoi any real resampling, we have line this problem with finite group action an converte it into a problem of orbit enumeration. For further computational cost reuction, an efficient metho is evelope to list all orbits by their symmetry orers an calculate all inex function orbit sums an ata function orbit sums recursively. The computational complexity analysis shows reuction in the computational cost from n! or n n level to low-orer polynomial level. Introuction Resampling methos (e.g., bootstrap, cross-valiation, an permutation) [3,5] are becoming increasingly popular in statistical analysis ue to their high flexibility an accuracy. They have been successfully integrate into most research topics in machine learning, such as feature selection, imension reuction, supervise learning, unsupervise learning, reinforcement learning, an active learning [2, 3, 4, 7, 9,, 2, 3, 20]. The ey iea of resampling is to generate the empirical istribution of a test statistic by resampling with or without replacement from the original observations. Then further statistical inference can be conucte base on the empirical istribution, i.e., resampling istribution. One of the most important problems in resampling is calculating resampling statistics, i.e., the expecte values of test statistics uner the resampling istribution, because resampling statistics are compact representatives of the resampling istribution. In aition, a resampling istribution may be approximate by a parametric moel with some resampling statistics, for example, the first several moments of a resampling istribution [5, 6]. In this paper, we focus on computing resampling weighte v- statistics [8] (see Section 2 for the formal efinition). Suppose our ata inclues n observations, a weighte v-statistic is a summation of proucts of ata function terms an inex function terms, i.e., weights, over all possible observations chosen from n observations, where is the orer of the weighte v-statistic. If we treat our ata as points in a multi-imensional space, a weighte v-statistic can be consiere as an average of all possible weighte -points istances. The higher, the more complicate interactions among observations can be moele in the weighte v-statistic. Machine learning researchers have alreay use weighte v-statistics in hypothesis testing, ensity estimation, epenence measurement, ata pre-processing, an classification [6, 4, 9, 2]. Traitionally, estimation of resampling statistics is solve by ranom sampling since exhaustive examination of the resampling space is usually ill avise [5,6]. There is a traeoff between accuracy an computational cost with ranom sampling. To ate, there is no systematic an efficient solution to the issue of exact calculation of resampling statistics. Recently, Zhou et.al. [2] propose a recursive metho to erive moments of permutation istributions (i.e., empirical istribution generate by resampling without replacement). The ey strategy is to ivie the whole inex set (i.e., inices of all possible observations ) into several permutation equivalent inex subsets such that the summa-

2 tion of the ata/inex function term over all permutations is invariant within each subset an can be calculate without conucting any permutation. Therefore, moments are obtaine by summing up several subtotals. However, methos for listing all permutation equivalent inex subsets an calculating of the respective carinalities were not emphasize in the previous publication [2]. There is also no systematic way to obtain coefficients in the recursive relationship. Even only for calculating the first four moments of a secon orer resampling weighte v statistic, hunres of inex subsets an thousans of coefficients have to be erive manually. The manual erivation is very teious an error-prone. In aition, Zhou s wor is limite to permutation (resampling without replacement) an is not applicable to bootstrapping (resampling with replacement) statistics. In this paper, we propose a novel an computationally fast algorithm for computing weighte v- statistics in resampling both univariate an multivariate ata. In the propose algorithm, the calculation of weighte v-statistics is consiere as a summation of proucts of ata function terms an inex function terms over a high-imensional inex set an all possible resamplings with or without replacement. To avoi any resampling, we lin this problem with finite group actions an convert it into a problem of orbit enumeration [0]. For further computational cost reuction, an efficient metho has been evelope to list all orbits by their symmetry orer an to calculate all inex function orbit sums an ata function orbit sums recursively. With computational complexity analysis, we have reuce the computational cost from n! or n n level to low-orer polynomial level. Detaile proofs have been inclue in the supplementary material. In comparison with previous wor [2], this stuy gives a theoretical justification of the permutation equivalence partition iea an extens it to other types of resamplings. We have built up a soli theoretical framewor that explains the symmetry of resampling statistics using a prouct of several symmetric groups. In aition, by associating this problem with finite group action, we have evelope an algorithm to enumerate all orbits by their symmetry orer an generate a recursive relationship for orbits sum calculation systematically. This is a critical improvement which maes the whole metho fully programmable an frees ourselves from onerous erivations in [2]. 2 Basic iea In general, people prefer choosing statistics which have some symmetric properties. All resampling strategies, such as permutation an bootstrap, are also more or less symmetric. These facts motivate us to reuce the computational cost by using abstract algebra. This stuy is focuse on computing resampling weighte v-statistics, i.e., T (x) = P n P i = n i = w(i,,i )h(x i, x i ), where x =(x,x 2,,x n ) T is a collection of n observations (univariate/multivariate), w is an inex function of inices, an h is a ata function of observations. Both w an h are symmetric, i.e., invariant uner permutations of the orer of variables. Weighte v-statistics cover a large amount of popular statistics. For example, in the case of multiple comparisons, observations are collecte from g groups: first group (x,,x n ), secon group (x n+,,x n+n 2 ), an last group (x n ng+,,x n ), where n,n 2,,n g are numbers of observations in each group. In orer to test the ifference among groups, it is common to use the moifie F test statistic T (x) =( P n i= x i) 2 /n +( P n +n 2 i=n x + i) 2 /n 2 + +( P n i=n n x g+ i) 2 /n g, where n = n + n n g. We can rewrite the moifie F statistic [3] as a secon orer weighte v-statistic, i.e., T (x) = P n P n i = i w(i 2=,i 2 )h(x i,x i2 ), here h(x i,x i2 )=x i x i2 an w(i,i 2 )=/n if both x i an x i2 belong to the -th group, an w(i,i 2 )=0otherwise. The r-th moment of a resampling weighte v-statistic is: r E T r (x) = E w(i,,i )h(x i,,x i ) i,,i n ry ry o = E w(i,,i ) h(x i,,x i ) = R i,,i,,ir,,ir 2R = i,,i,,ir,,ir n ry = = ry w(i,,i ) = o h(x i,,x i ), () 2

3 where is a resampling which is uniformly istribute in the whole resampling space R. R, the number of all possible resamplings, is equal to n! or n n for resampling without or with replacement. Thus the r-th moment of a resampling weighte v-statistic can be consiere as a summation of proucts of ata function terms an inex function terms over a high-imensional inex set U r = {,,n} r an all possible resamplings in R. Since both inex space an resampling space are huge, it is computationally expensive for calculating resampling statistics irectly. For terminology convenience, {(i,,i ),, (ir,,i r )} is calle an inex paragraph, which inclues r inex sentences (i,,i ), =,,r, an each inex sentence has inex wors i j,j =,,. Note that there are three ifferent types of symmetry in computing resampling weighte v-statistics. The first symmetry is that permutation of the orer of inex wors will not affect the result since the ata function is assume to be symmetric. The secon symmetry is the permutation of the orer of inex sentences since multiplication is commutative. The thir symmetry is that each possible resampling is equally liely to be chosen. In orer to reuce the computational cost, first, the summation orer is exchange, E T r (x) = i,,i,,ir,,ir n ry = w(i,,i ) E ry = o h(x i,,x i ), (2) where E = h(x i,,x i ) = P R 2R = h(x i,,x i. ) n The whole inex set U r = {,,n} r = {(i,,i ),, (ir,,i r )} i m 2 o {,,n}; m =,,; =,,r is then ivie into isjoint inex subsets, in which E = h(x i,, x i ) is invariant. The above inex set partition simplifies the computing of resampling statistics in the following sense: (a) we only nee to calculate Qr E = h(x i,,x i ) once per each inex subset, (b) ue to the symmetry of resampling, the calculation of E = h(x i,,x i ) is equivalent to calculating the average of all ata function terms within the corresponing inex subset, then we can completely replace all resamplings with simple summations, an (c) for further computational cost reuction, we can sort all inex subsets in their symmetry orer an calculate all inex subset summations recursively. We will iscuss the etails in the following sections for both resampling without or with replacement. The abstract algebra terms use in this paper are liste as follows. Terminology. A group is a non-empty set G with a binary operation satisfying the following axioms: closure, associativity, ientity, an invertibility. The symmetric group on a set, enote as S n, is the group consisting of all bijections or permutations of the set. A semigroup has an associative binary operation efine an is close with respect to this operation, but not all its elements nee to be invertible. A monoi is a semigroup with an ientity element. A set of generators is a subset of group elements such that all the elements in the group can be generate by repeate composition of the generators. Let be a set an G be a group. A group action is a mapping G! which satisfies the following two axioms: (a) e x 7! x for all x 2, an (b) for all a, b 2 G an x 2, a (b x) =(ab) x. Here the 0 0 enotes the action. It is well nown that a group action efines an equivalence relationship on the set, an thus provies a isjoint set partition on it. Each part of the set partition is calle an orbit that enotes the trajectory move by all elements within the group. We use symbol []to represent an orbit. Two elements, x an y 2 fall into the same orbit if there exists a g 2 G such that x = g y. The set of orbits is enote by G. A transversal of orbits is a set of representatives containing exactly one element from each orbit. In this paper, we limit our iscussion to only finite groups [0,7]. 3 Permutation For permutation statistics, observations are permute in all possible ways, i.e., R = S n. Base on the three types of symmetry, we lin the permutation statistics calculation with a group action. Definition. The action of G := S n S r S r on the inex set U r is efine as 3

4 (,,,, r ) i m := i m, where m 2{,,}, an 2{,,r}. Here, enotes the permutation of the orer of inex wors within the -th inex sentence, enotes the permutation of the orer of r inex sentences, an enotes the permutation of the value of an inex wor from to n. For example, let n =4, =2, r =2, = =! 2, 2!, 2 = 2 =!, 2! 2, = =! 2, 2!, an =! 2, 2! 4, 3! 3, 4!, then (,,, 2 ) {(, 4)(3, 4)} = {(3, )(, 2)} by {(, 4)(3, 4)}!{(4, )(3, 4)}! {(3, 4)(4, )}!{(3, )(, 2)}. Note that the reason to efine the action in this way is to guarantee G U r! U r is a group action. In most applications, both r an are much less than the sample size n, we assume throughout this paper that n r. Proposition. The ata function sum E = h(x i,,x i ) is invariant within each inex orbit of group action G := S n S r S r acting on the inex set U r as efine in efinition Qr, an E = h(x i,,x i = ) Q r = h(x j,,x j ), (3) {(j,,j ),,(jr,,jr )}2[{(i,,i ),,(ir,,ir )}] car [{(i,,i ),, (ir,,ir )}] where car [{(i,,i ),, (ir,,i r )}] is the carinality of the inex orbit, i.e., the number of inices within the inex orbit [{(i,,i ),, (ir,,i r )}]. Due to the invariance property of E = h(x i,,x i, ) the calculation of permutation statistics can be simplifie by summing up all inex function prouct terms in each inex orbit. Proposition 2. The r-th moment of permutation statistics can be obtaine by summing up the prouct of the ata function orbit sum h an the inex function orbit sum w over all inex orbits, E T r (x) = w h car([ ]), (4) 2L where = {(i,,i ),, (ir,,i r )} is a representative inex paragraph, [ ] is the inex orbit incluing, an L is a transversal of all inex orbits. The ata function orbit sum is ry h = h(x j,,x j ), (5) an the inex function orbit sum is w = {(j,,j ),,(jr,,jr )}2[ ] = {(j,,j ),,(jr,,jr )}2[ ] = ry w(j,,j ). (6) Proposition 2 shows that the calculation of resampling weighte v-statistics can be solve by computing ata function orbit sums, inex function orbit sums, an carinalities of all orbits efine in efinition. We on t nee to conuct any real permutation at all. Now we emonstrate how to calculate orbit carinalities, h an w. The following shows a naive algorithm to enumerate all inex paragraphs an carinality of each orbit of G U r, which are neee to calculate h an w. We construct a Cayley Action Graph with a vertex set of all possible inex paragraphs in U r. We connect a irecte ege from {(i,,i ),, (ir,,i r )} to {(j,,j ),, (jr,,j r)} if {(j,,j ),, (j, r,j r)} = g {(i,,i ),, (ir,,i r )}, where g is a generator 2{g,,g p }. {g,,g p } is the set of generators of group G, i.e., G = hg,,g p i. It is sufficient an efficient to use the set of generators of group to construct the Cayley Action Graph, instea of using the set of all group elements. For example, we can choose {g,,g p } = {, 2} {, 2 } {, 2 } r, where = (2 n), 2 = (2), = (2 r), 2 = (2), = (2 ), an 2 = (2). Here = (2 n) enotes the permutation! 2, 2! 3,,n!, an 2 = (2) enotes 4

5 ! 2, 2!, 3! 3,, n! n. Note that listing the inex paragraphs of each orbit is equivalent to fining all connecte components in the Cayley Action Graph, which can be performe by using existing epth-first or breath-first search methos [5]. Figure emonstrates the Cayley Action Graph of G U2, where =2, r =, an n =3. Since the main effort here is to construct the Cayley Action Graph, the computational cost of the naive algorithm is O(n r p)=o(n r 2 2+r ). Moreover, the memory cost is O(n r ). Unfortunately, this algorithm is not an offline one since we usually o not now the ata size n before we have the ata at han, even an r can be preset. In other wors, we can not list all inex orbits before we now the ata size n. Moreover, since n r 2 2+r is still computationally expensive, the naive algorithm is ill avise even if n is preset. i r G \ \U * G \ \ r* U i 2 3 Cayley action graph Set of orbits U 2 [{(, )}] [{(, 2)}] transversal Figure : Cayley action graph for G U2. Figure 2: Fining the transversal. In table, we propose an improve offline algorithm in which we assume that an r are preset. For computing h an w, we fin that we o not nee to now all the inex paragraphs within each inex orbit. Since each orbit is well structure, it is enough to only list a transversal of orbits G U r an corresponing carinalities. For example, there are two orbits, [{(, )}] an [{(, 2)}], when =2an r =. [{(, )}], with carinality n, inclues all inex paragraphs with i = i 2. [{(, 2)}], with carinality n n(n ), o inclues all inex paragraphs with i 6= i 2. Actually, the transversal L = {(, )}, {(, 2)} carries all the above information. This fining reuces the computation cost ramatically. n Definition 2. We efine an inex set U r = {,,r} r = {(i,,i ),, (ir,,i r )} i m o 2{,,r}; m =,,; =,,r an a group G := S r S r S r. Since we assume n r, U r is a subset of the inex set U r. The group G can be consiere a subgroup of G since the group S r can be naturally embee into the group S n. Both U r an G are unrelate to the sample size n. Proposition 3. The transversal of G U r is also a transversal of G U r. By proposition 3, we notice that the listing of the transversal of G U r is equivalent to the listing of the transversal of G U r (see Figure 2). The latter is computationally much easier than the former since the carinalities of G an U r are much smaller than those of G an U r when n r. Furthermore, fining the transversal of G U r can be one without nowning sample size n. Due to the structure of each orbit of G U r, we can calculate the carinality of each orbit of G U r with the transversal of G U r, although G U r an G U r have ifferent carinalities for corresponing orbits. Table : Offline ouble sie searching algorithm for listing the transversal Input: an r,. Starting from an orbit representative {(,,),, ((r ) +,,r)} 2. Construct the transversal of S r U r by merging 3. Construct the transversal of of G U r by graph isomorphism testing 4. Ening to an orbit representative {(,, ),, (,, )} Output: a transversal L of G U r, #( ), #(! ), an merging orer(symmetry orer) of orbits Comparing with the Cayley Action Graph naive algorithm, our improve algorithm lists the transversal of G U r an calculates the carinalities of all orbits more efficiently. In aition, the improve algorithm also assigns a symmetry orer to all orbits, which helps further reuce the computational 5

6 cost of the ata function orbit sum h an the inex function orbit sum w. The base of our improve algorithm is on the fact that a subgroup acting on the same set causes a finer partition. On one han, it is challenging to irectly list the transversal of G U r. On the other han, it is much easier to fin two relate group actions, causing finer an coarser partitions of U r. These two group actions help us fin the transversal of G U r efficiently with a ouble sie searching metho. Definition 3. The action of S r on the inex set U r is efine as i m, where 2 S r, m 2 {,,}, an 2 {,,r}. Each orbit of S r U r is enote by [{(i,,i ),, (ir,,i r )}]s. Note the group action efine in efinition 3 only allows permutation of inex values, it oes not allow shuffling of inex wors within each inex sentence or of inex sentences. Since S r is embee in G, the set of orbits S r U r is a finer partition of G U r. For example, both [{(, 2)(, 2)}] s an [{(, 2)(2, )}] s are finer partitions of [{(, 2)(, 2)}]. In aition, it is easy to construct a transversal of S r U r by merging istinct inex elements. Definition 4. Given a representative I, which inclues at least two istinct inex values, for example i 6= j, an operation calle merging replaces all inex values of i or j with min(i, j). For example, [{(, 2)(2, 3)}] becomes [{(, )(, 3)}] after merging the inex values of an 2. Definition 5. The action of S r S r on the inex set U r is efine as i (,m) s (,m) w, where 2 S r enotes a permutation of all r inex wors without any restriction, i.e. (, m) s enotes the inex sentence location after permutation, an (, m) w enotes the inex wor location after permutation. The orbit of S r S r U r is enote by [{(i,,i ),, (ir,,i r )}]l. Since the group action efine in efinition 5 allows free shuffling of the orer of all r inex wors, the orer oes not matter for S r S r U r an shuffling can across ifferent sentences. For example, [{(, 2)(, 2)}] l =[{(, )(2, 2)}] l. S r S r is a coarser partition of G. Proposition 4. A transversal of S r U r can be generate by all possible mergings of [{(,,),, ((r ) +,,r)}] s. Proposition 5. Enumerating a transversal of S r S r U r is equivalent to the integer partition of r. We start the transversal graph construction from an initial orbit [{(,,),, ((r ) +,,r)}] s, i.e, all inex elements have istinct values. Then we generate new orbits of S r U r by merging istinct inex values in existing orbits until we meet [{(,, ),, (,, )}] s, i.e., all inex elements have equal values. We also a an ege from an existing orbit to a new orbit generate by merging the existing one. The proceure for =2, r =2case is shown in Figure 3. Now we generate the transversal of G U r from that of S r U r. This can be one by checing whether two orbits in S r U r are equivalent in G U r. Actually, orbit equivalence checing is equivalent to the classical graph isomorphism problem since we can consier each inex wor as a vertex an connect two inex wors if they belong to the same inex sentence. The graph isomorphism testing can be one by Lus s famous algorithm [,5] with computational cost exp O( p vlogv), where v is the number of vertices. Figure 4 shows a transversal of G U2 2 generate from that of S 4 U2 2 (Figure 3). By proposition 3, it is also a transversal of G U 2 2. Since G U r is a finer partition of S r S r U r, orbit equivalence testing is only necessary when two orbits of S r U r correspon to the same integer partition. This is why we name this algorithm ouble sie searching. U r U r [{(,2)(3,4)}] s [{(,2)(3,4)}] [{(,)(3,4)}] s [{(,2)(,4)}] s [{(,2)(3,)}] s [{(,2)(2,4)}] s [{(,2)(3,2)}] s [{(,2)(3,3)}] s [{(,)(,4)}] s [{(,)(3,)}] s [{(,)(3,3)}] s [{(,2)(,)}] s [{(,2)(,2)}] s [{(,2)(2,)}] s [{(,2)(2,2)}] s [{(,)(,)}] s Figure 3: Transversal graph for S 4 U 2 2. [{(,)(,2)}] [{(,)(2,3)}] [{(,2)(,3)}] [{(,)(2,2)}] [{(,2)(,2)}] [{(,)(,)}] Figure 4: Transversal graph for G U

7 Definition 6. For any two inex orbit representatives 2 L an 2 L, we say that has a lower merging or symmetry orer than that of, i.e.,, if [ ] can be obtaine from [ ] by several mergings. Or there is a path from [ ] to [ ] in the transversal graph. Here L enotes a transversal set of all orbits. Definition 7. We efine #( ) as the number of S r U r orbits in [ ]. We also efine #(! ) as the number of ifferent [ ] s s which can be reache from a [ ] s. It is easy to get #( ) when we generate a transversal graph of G U r from that of S r U r. The #(! ) can also be obtaine from the transversal graph of G U r by counting the number of ifferent [ ] s s which can be reache from a [ ] s. For example, there are eges connecting [{(, )(3, 4)}] s to [{(, )(, 4)}] s an [{(, )(3, )}] s. Since [{(, )(, 4)}] =[{(, )(3, )}] = [{(, )(, 2)}], #( = {(, )(2, 3)}! = {(, )(, 2)}) =2. Note that this number can also be obtaine from [{(, 2)(3, 3)}] s to [{(, 2)(, )}] s an [{(, 2)(2, 2)}] s. The ifficulty for computing ata function orbit sum an inex function orbit sum comes from two constraints: equal constraint an unequal constraint. For example, in the orbit [{(, ), (2, 2)}], the equal constraint is that the first an the secon inex values are equal an the thir an fourth inex values are also equal. On the other han, the unequal constraint requires that the first two inex values are ifferent from the last two. Due to the ifficulties mentione, we solve this problem by first relaxing the unequal constraint an then applying the principle of inclusion an exclusion. Thus, the calculation of an orbit sum can be separate into two parts: the relaxe orbit sum without unequal constraint an lower orer orbit sums. For example, the relaxe inex function orbit sum is w =[{(,),(2,2)}] = P i,j w(i, i)w(j, j) = P i w(i, i) 2. Proposition 6. The inex function orbit sum w can be calculate by subtracting all lower orer orbit sums from the corresponing relaxe inex function orbit sum w, i.e., w = w #( ) P w #( ) #(! ). The carinality of [ ] is #( )n(n ) (n q + ), where q is the number of istinct values in. The calculation of the ata inex function orbit sum h is similar. So the computational cost mainly epens on the calculation of relaxe orbit sum an the lowest orer orbit sum. The computational cost of the lowest orer term is O(n). The calculation of relaxe orbit can be one by Zhou s greey graph search algorithm [2]. Proposition 7. For 2, let m(m )/2 apple r( )/2 < (m + )m/2, where r is the orer of moment an m is an integer. For a -th orer weighte v-statistic, the computational cost of the orbit sum for the r-th moment is boune by O(n m ). When =, the computational complexity of the orbit sum is O(n). 4 Bootstrap Since Bootstrap is resamping with replacement, we nee to change S n to the set of all possible enofunctions En n in our computing scheme. In mathematics, an enofunction is a mapping of a set to its subset. With this change, H := En n S r S r acting on U r becomes a monoi action instea of a group action since enofunction is not invertible. The monoi action also ivies the U r into several subsets. However, these subsets are not necessarily isjoint after mapping. For example, when =2an r =, we can still ivie the inex set U2 into two subsets, i.e., [(, )] an [(, 2)]. However, [(, 2)] is mappe to U2 =[(, 2)] S [(, )] by monoi action H U r! U r, although [(, )] is still mappe to itself. Fortunately, the computation of Bootstrap weighte v-statistics only nees inex function orbit sums an relaxe ata function orbit sums in the corresponing permutation computation. Therefore, the Bootstrap weighte v-statistics calculation is just a subproblem of permutation weighte v-statistics calculation. Proposition 8. We can obtain the r-th moment of bootstrapping weighte v-statistics by summing up the prouct of the inex function orbit sum w an the relaxe ata function orbit sum h over all inex orbits, i.e., E (T r (x)) = w h car([ ]), (7) 2L where 2 En n, car([ ]) = #( )n q, an q is the number of istinct values in. 7

8 Table 2: Comparison of accuracy an complexity for calculation of resampling statistics. Permutation Bootstrap Linear Quaratic Linear Quaratic Methos 2n moment 3r 4th Time Exact e3 Our Ranom Exact.06e e e6.78e3 Our.06e e e Ranom.0569e e e Exact Our Ranom Exact e e e Our e e e Ranom e e e8.987 The computational cost of bootstrapping weighte v-statistics is the same level as that of permutation statistics. 5 Numerical results To evaluate the accuracy an efficiency of our moths, we generate simulate ata an conuct permutation an bootstrapping for both linear test statistic P n P i= w(i)h(x i) an quaratic test statistic n P n i = i w(i 2=,i 2 )h(x i,x i2 ). To emonstrate the universal applicability of our metho an prevent a chance result, we generate w(i),h(x i ),w(i,i 2 ),h(x i,x i2 ) ranomly. We compare the accuracy an complexity among exact permutation/bootstrap, ranom permutaton/bootrap (0,000 times), an our methos. Table 2 shows comparisons for computing the secon, thir, an fourth moments of permutation statistics with observations (the running time is in secons) an of bootstrap statistics with 8 observations. In all cases, our metho achieves the same moments as those of exact permutation/bootstrap, an reuces computational cost ramatically comparing with both ranom sampling an exact sampling. For emonstration purpose, we choose a small sample size here, i.e., sample size is for permutation an 8 for bootstrap. Our metho is expecte to gain more computational efficiency as n increases. 6 Conclusion In this paper, we propose a novel an computationally fast algorithm for computing weighte v- statistics in resampling both univariate an multivariate ata. Our theoretical framewor reveals that the three types of symmetry in resampling weighte v-statistics can be represente by a prouct of symmetric groups. As an exciting result, we emonstrate the calculation of resampling weighte v-statistics can be converte into the problem of orbit enumeration. A novel efficient orbit enumeration algorithm has been evelope by using a small group acting on a small inex set. For further computational cost reuction, we sort all orbits by their symmetry orer an calculate all inex function orbit sums an ata function orbit sums recursively. With computational complexity analysis, we have reuce the computational cost from n! or n n level to low-orer polynomial level. 7 Acnowlegement This research was supporte by the Intramural Research Program of the NIH, Clinical Research Center an through an Inter-Agency Agreement with the Social Security Aministration, the NSF CNS 35660, Office of Naval Research awar N , Air Force Office of Scienticfic Research awar FA , an IC Postoctoral Research Fellowship awar

9 References [0] Babai, L., Kantor, W.M., an Lus, E.M. (983), Computational complexity an the classification of finite simple groups, Proc. 24th FOCS, pp [02] Minaei-Bigoli, B., Topchy, A., an Punch, W. (2004), A comparison of resampling methos for clustering ensembles, In Proc. International Conference on Artificial Intelligence, Vol. 2, pp [03] Estabroos, A., Jo, T., an Japowicz, N. (2004), A Multiple Resampling Metho for Learning from Imbalance Data Sets, Comp. Intel. 20 () pp [04] Francois, D., Rossib, F., Wertza, V., an Verleysen, M. (2007), Resampling methos for parameter-free an robust feature selection with mutual information, Neurocomputing 70(7-9): [05] Goo, P. (2005), Permutation, Parametric an Bootstrap Tests of Hypotheses, Springer, New Yor. [06] Gretton, A., Borgwart, K., Rasch, M., Scholopf, B., an Smola, A. (2007), A ernel metho for the two-sample- problem, In Avances in Neural Information Processing Systems (NIPS). [07] Guo, S. (20), Bayesian Recommener Systems: Moels an Algorithms, Ph.D. thesis. [08] Hopcroft, J., an Tarjan, R. (973), Efficient algorithms for graph manipulation, Communications of the ACM 6: [09] Huang, J., Guestrin, C., an Guibas, L. (2007), Efficient Inference for Distributions on Permutations, In Avances in Neural Information Processing Systems (NIPS). [0] Kerber, A. (999), Applie Finite Group Actions, Springer-Verlag, Berlin. [] Konor, R., Howar, A., an Jebara, T. (2007), Multi-Object Tracing with Representations of the Symmetric Group, Artificial Intelligence an Statistics (AISTATS). [2] Kuwaear, A. an Neville, J. (20), Relational Active Learning for Joint Collective Classification Moels, In International Conference on Machine Learning (ICML), P [3] Liu, H., Palatucci, M., an Zhang, J.(2009), Blocwise coorinate escent proceures for the multi-tas lasso, with applications to neural semantic basis iscovery, In International Conference on Machine Learning (ICML). [4] Matthew Higgs an John Shawe-Taylor. (200), A PAC-Bayes boun for tailore ensity estimation, In Proceeings of the International Conference on Algorithmic Learning Theory (ALT). [5] McKay, B. D. (98), Practical graph isomorphism, Congressus Numerantium 30: 45-87, 0th. Manitoba Conf. on Numerical Math. an Computing. [6] Miele, P. W., an K. J. Berry (2007), Permutation Methos: A Distance Function Approach, Springer, New Yor. [7] Nicholson, W. K. (2006), Introuction to Abstract Algebra, 3r e., Wiley, New Yor. [8] Serfling, R. J. (980), Approximation Theorems of Mathematical Statistics, Wiley, New Yor. [9] Song, L. (2008), Learning via Hilbert Space Embeing of Distributions, Ph.D. thesis. [20] Sutton, R. an Barto, A. (998), Reinforcement Learning, MIT Press. [2] Zhou, C., Wang, H., an Wang, Y. M. (2009), Efficient moments-base permutation tests, In Avances in Neural Information Processing Systems (NIPS), p