A Variance Minimization Criterion to Active Learning on Graphs

Transcription

1 A Variance inimization Criterion to Active Learning on Grahs ing Ji Deartment of Comuter Science University of Iinois at Urbana-Chamaign Jiawei Han Deartment of Comuter Science University of Iinois at Urbana-Chamaign Abstract We consider the robem of active earning over the vertices in a grah, without feature reresentation. Our study is based on the common grah smoothness assumtion, which is formuated in a Gaussian random fied mode. We anayze the robabiity distribution over the unabeed vertices conditioned on the abe information, which is a mutivariate norma with the mean being the harmonic soution over the fied. hen we seect the nodes to abe such that the tota variance of the distribution on the unabeed data, as we as the exected rediction error, is minimized. In this way, the cassifier we obtain is theoreticay more robust. Comared with existing methods, our agorithm has the advantage of seecting data in a batch offine mode with soid theoretica suort. We show imroved erformance over existing abe seection criteria on severa rea word data sets. Introduction In many domains of interest, data instances are connected by edges reresenting certain reationshis, forming a grah structure. Grahs and feature vectors are two aternatives to reresent the data, and the former is often more natura than the atter in many data sets [9] incuding eoe inked by the friendshi reation in socia networks, web ages interconnected by hyerinks, etc. Even if the origina data has feature reresentation, it is usuay hefu to transform the data into a grah structure (via constructing a nearest neighbor grah, for instance) to better exoit Aearing in Proceedings of the 5 th Internationa Conference on Artificia Inteigence and Statistics (AISAS) 202, La Pama, Canary Isands. Voume XX of JLR: W&CP XX. Coyright 202 by the authors. roerties of the data. In this way, earning on grahs is receiving more and more attention in recent years. Substantia efforts have been devoted to the robem of cassification of the nodes in a grah. On the other hand, abes can be very exensive to obtain in many rea-word aications. Active earning [7] is then roosed to determine which data exames shoud be abeed such that the cassifier coud achieve higher rediction accuracy over the unabeed data as comared to random abe seection. he goa of active earning is to maximize the earner s abiity given a fixed budget of abeing effort. Whie many effective active earners have been deveoed in iterature [20], active earning that takes direct advantage of the grah structure in the data has not been exored unti recenty [9, 2, 25]. As arge-scae data sets with inherent grah structures become increasingy revaent, reasonabe and natura active earning criteria on grahs are in great demand. ost of the existing active earners work with data reresented by feature vectors [20]. In a semina aer [2], X. He rooses the first manifod-based active earning agorithm, i.e., LaRDD, which takes into account both the discriminant and geometrica structure in the data. A nearest neighbor grah is constructed to mode the intrinsic manifod structure and incororated into a east squares oss function as a reguarizer. he most informative data oints are seected by minimizing the size of the arameter covariance matrix. his rincie has been successfuy aied to image retrieva [2], video indexing [22], and feature seection [3]. Pease see [2] for another active earning aroach that exoits the features together with the grah structure. However, in some cases, features of the grah nodes are not aways avaiabe. Some other methods try to seect data based on the grah structure and some abeed nodes. Existing aroaches have considered seecting the data that the current cassifier is the most uncertain [7], the data with maximum exected information gain [23] or maximum exected entroy reduction [6]. Based on the Gaussian random fied mode [24], an emirica risk minimization framework [25] is roosed to seect ex- 556

2 A Variance inimization Criterion to Active Learning on Grahs ames that minimize the emirica risk estimated by the current cassifier. One major imitation of these methods [25, 7, 4, 23, 6, 5] is that they have to obtain the abes of the seected nodes in order to seect more data, therefore are not aicabe when there is no abe information rovided during active earning. When abeing an instance requires time consuming and exensive exeriments, these methods are much more costy than running a batch offine mode active earner once and erform abeing in arae [9]. Recenty, there are some efforts devoted to designing abe seection criteria that use the grah structure ony, without feature reresentation and abe information. Intuitivey, one tends to seect nodes that ie in high-density (unabeed) regions [5] or the centers of custers [7], or have high imact (measured by the grah structure) to unabeed data [2]. However, these intuitive seection criteria do not have theoretica suort on otimizing any cassifier. In this aer, we roose a nove variance minimization ersective to active earning urey on the grah structure, without feature reresentation and abe information. Our study is based on the common assumtion that the abes vary smoothy with resect to the grah, which is widey used in the grah-based semisuervised earning iterature [5, 3, 0, 9, ]. Foowing one of the most ouar grah-based earning frameworks [24], we formuate the smoothness assumtion by a Gaussian random fied over the grah nodes. heoretica anaysis indicates that the Gaussian fied over the unabeed vertices, conditioned on the abeed data, is a mutivariate norma whose mean is the rediction of the harmonic Gaussian fied cassifier [24]. It is interesting to note that the covariance matrix of the Gaussian fied over the unabeed data is not deendent on the cass abes, but ony on the grah structure. In this way, we roose to seect the data oints to abe such that the tota variance of the Gaussian fied over unabeed exames, as we as the exected rediction error of the harmonic Gaussian fied cassifier, is minimized. Efficient comutation scheme is then roosed to sove the corresonding otimization robem without introducing any additiona arameter. In fact, designing active earners on grahs aiming at minimizing the error of a articuar cassifier has received substantia interest recenty [9, 25]. [9] rovides theoretica bounds of the rediction error which are reated to abe smoothness over the grah, justifying the reasonabeness of custering the nodes and then randomy choose one oint from each custer. Comared with existing methods [9, 25, 2, 5], our agorithm has the advantage of directy minimizing the exected error (instead of the uer bound of the error) in a batch 557 offine mode, through reasonaby modeing the robabiity distribution over the grah. herefore, we do not require the (otentiay exensive) abe information of the seected data and tedious retraining of the cassifier reeatedy. he rest of this aer is organized as foows. In the next section, we introduce the variance minimization ersective for active earning on grahs. Section 3 resents a sequentia otimization scheme that efficienty soves our objective function. Extensive exerimenta resuts on three rea-ife data sets are resented in Section 4. We rovide some concuding remarks as we as suggestions for future work in Section 5. 2 A Variance inimization Criterion to Active Learning on Grahs 2. he Probem We define the active earning robem on grahs as foows. Given a grah G = V, E associated with a weight matrix W, where V = {v,..., v n } is the set of data oints (without feature reresentation) with true abes y = (y,..., y n ), E is the set of edges between any two data oints in V, and W = (w ij ) R n n where w ij denotes the weight on the edge between two data oints v i and v j. Our goa is to find a subset of oints L = {v,..., v } V where { i } {,..., n} are the indices of the oints that we shoud abe, such that the cassifier earned from the abes on L coud achieve the smaest exected rediction error on the unabeed data, measured by v (y i U i yi )2, where U = V \ L and yi is the redicted abe for v i. Without oss of generaity, in this aer, we assume that G is undirected and connected. We aow continuous abes here, and the abes are assumed to vary smoothy over the grah, i.e., i,j w ij(y i y j ) 2 is sma, which is simiar to [9]. 2.2 he Objective Function Foowing [24, 25], the abe smoothness assumtion coud be formuated by a Gaussian random fied over the grah: P (y) = ex( βe(y)) () Z β where E(y) = 2 i,j w ij (y i y j ) 2 is the energy function measuring the smoothness of a abe assignment y = (y,..., y n ) over the grah, β is an inverse temerature arameter, and Z β is a artition function for the normaization urose. Without oss of generaity, we can arrange the data oints chosen to be abeed to be the first instances,

3 ing Ji, Jiawei Han i.e., L = {v,..., v }, and the rest u(= n ) exames U = {v +,..., v +u } are unabeed. Based on the Gaussian random fied mode, and the constraint that the redictions on the abeed set are consistent with ground truth, i.e., yl = y L = (y,..., y ), a standard method is to redict the abes with the highest robabiity (or equivaenty, minimum energy) [24, 25]. Let L = D W be the grah Laacian [6], where D is a diagona matrix and D ii = j w ij. L can be sit into 4 bocks according to the -th row and coumn: L L L = u (2) L u L uu hen the rediction on the unabeed nodes given by the harmonic Gaussian fied cassifier is [24]: where y U = (y +,..., y +u ). y U = L uu L u y L (3) It can be roven that the Gaussian fied, conditioned on the abeed data, is a mutivariate norma: y U N (yu, L uu ) [25], where y U = (y +,..., y +u ). hen we comute the exected rediction error on the unabeed nodes as foows: E (y i yi ) 2 v i U = E ( (y U y U) (y U y U) ) = E ( r ( (y U y U)(y U y U) )) = r ( E ( (y U y U)(y U y U) )) = r (var(y U )) = r(l uu ) (4) In order to minimize the exected error of the rediction resuts, we shoud minimize the variance of the statistica earning mode [7]. herefore, we roose to seect the nodes to abe by soving the foowing otimization robem: the number of candidate sets for L is exonentia in the tota number of exames n. oreover, since the number of unabeed exames is usuay huge, L uu wi ikey be a arge matrix and directy otimizing Eq. (5) based on the set of unabeed data is very comutationay exensive. In this section, we first transform the objective function so that it can be reresented by the instances that we choose to abe, and then roose an efficient sequentia otimization scheme. 3. Formuations We first construct a seection matrix S R u n to he seecting L uu from L as foows: hen we have: S ij = { if j = qi 0 otherwise. L uu = SLS (6) Since L is symmetric, it has the eigendecomosition resut as foows: L = XΣX (7) such that X is an orthonorma matrix, and Σ = diag {λ,..., λ n }, where {λ i } n are the eigenvaues of L, and λ... λ n = 0. hen L uu = SLS = SXΣX S (8) Suose X = (x,..., x n ), where x i is the i-th row of X. Let Q = SX, then L uu = QΣQ. Since S is the seection matrix, then Q = (q,..., q u ) R u n consists of the {q,..., q u }-th rows of X. We further define two sets of vectors X = {x,..., x n }, Q = {q,..., q u }, then our objective function in Eq. (5) is equivaent to the foowing: arg min r(l uu ) (5) L V It is easy to verify that Eq. (5) is indeendent of the order of the exames, but ony deendent on the choice of the set of the nodes that we choose not to abe. herefore, our objective function is we defined. 3 Efficient Otimization Let {q,..., q u } be the indices of the nodes that we choose not to abe. Foowing the above discussion, our objective is to seect a u u submatrix L uu of L on the intersections of the {q,..., q u }-th rows and coumns, such that the trace of L uu is minimized. his otimization robem in Eq. (5) is chaenging since 558 arg min r ( (QΣQ ) ) (9) Q X Let I n denote the identity matrix of size n n. By using the Woodbury formua [8], we have the foowing: (QΣQ ) = ( Q(Σ + I n )Q QQ ) = ( Q(Σ + I n )Q I u ) = ( I u ) Q ( (Σ + I n ) + Q ( I u ) Q ) Q = I u Q ( Q Q ) Q where = Σ + I n = diag {λ +,..., λ n + }. According to the matrix determinant emma [], we

4 A Variance inimization Criterion to Active Learning on Grahs have: det ( Q Q ) = () n det ( + Q Q ) = () n det ( ) det (I u + Q ( ) ) Q = () 2n det ( ) det ( I u QQ ) = det ( I u QΣQ QI n Q ) n = det (I u L uu I u ) = det( L uu ) n n λ i + λ i + λ i + (0) As ong as 0 < u < n and the grah is connected, it can be easiy roven that L uu is invertibe, and so is Q Q. Reca that r(ab) = r(ba), we further have: r ( (QΣQ ) ) = u r (Q ( Q Q ) ) Q ( ( = u r Q Q ) ) Q Q = u ( ( +r Q Q ) ) ( Q Q + ) = u + r (I n ( Q Q ) ) ( ( = n u r Q Q ) ) ) u = r q i q i Let P = {,..., } = X \ Q be the {,..., }-th row vectors of X that corresond to the exames that we choose to abe, then we have: r ( (QΣQ ) ) = r = r = r = r ) u q i q i n x i x i + X X + I n + ) i i ) i i ) i i Let A 0 = I n. Since the number of data oints to be abeed,, is fixed, our objective function in Eq. 559 (9) reduces to the foowing: arg max P X r A 0 + ) i i () In the foowing, we describe an efficient sequentia otimization scheme to seect which nodes we shoud abe in a grah. 3.2 Seecting the First Point Setting = in Eq. (), we obtain the objective function of seecting one (or the first) data oint to abe: ( (A0 arg max r + ) ) (2) X Usuay, matrix inversion formuae in the form of ( A0 + ) can be simified using the Sherman- orrison formua [8]: (A + uv ) = A A uv A + v A u However, note that A 0 = I { n } = diag λ +,..., λ n + { } λ = diag λ +,..., λ n λ n + (3) (4) is singuar since the smaest eigenvaue of L (denoted as λ n ) is equa to 0. herefore, the Sherman-orrison formua (3) cannot be aied here. In this subsection, we derive how to seect the first oint to abe by erforming some modification of Eq. (2). For a connected grah, it is known that a the eigenvaues of L, excet λ n, are arger than 0. he eigenvector corresonding to λ n is a n constant vector which can be denoted as (c,..., c). So any X can be reresented as = (v, c) where v is a (n ) vector after removing the ast eement of {. Let B = diag λ λ +,..., λn λ n+ } R (n) (n) be the matrix after removing the ast row and coumn of A 0, which is invertibe. Hence: = A 0 + ( B cv cv c 2 = ˆB + ˆvˆv ) ( v + 0 ) ( v 0 )

5 ing Ji, Jiawei Han where ˆB B cv = cv c 2 and ˆv = ( v 0 ). By doing bockwise matrix inversion, we have: ˆB B 0 = 0 0 c + 2 B vv B cb v c 2 ( v B v) cv B { where B = diag λ+ λ can emoy Eq. (3) and have:,..., λn+ λ n }. Now we Since r(a ) is a constant when seecting the ( + )-th data oint, we choose the ( + )-th oint to abe that corresonds to the foowing + : + = arg min X \P A A + A (20) Once + is obtained, A + can be udated according to Eq. (8). 4 Exerimenta Resuts = Reca that ( A0 + ) ( ˆB + ˆvˆv ) = ˆB ˆB ˆvˆv ˆB + ˆv ˆBˆv { = diag λ +,..., λ n + (5) }. herefore, ( A 0 + ) can be comuted efficienty without matrix inversion for any given. We seect the first data oint to abe that corresonds to X such that Eq. (2) is maximized. 3.3 Seecting ore Points We define: A = A 0 + i i (6) Suose ( ) data oints have been seected, which corresond to the rows of X: {,..., } = P X, then the ( + )-th instance can be seected by soving the foowing: ( (A + = arg max r + ) ) (7) X \P By using the Sherman-orrison formua (3), we have: ( A + ) = A A A + A (8) And A can be comuted using Eq. (5). herefore: ( (A r + ) ) = r(a = r(a = r(a ) r ( A A + A ) r ( A ) A ) A ) + A A + A (9) 560 In this section, we ay our roosed active earning method based on Variance inimization (denoted as ) in the Gaussian random fied to severa reaword data sets to test its effectiveness. We use the abes of vertices chosen by different active earning criteria to train a harmonic Gaussian fied cassifier [24] to redict the abes of the rest of the nodes in the grah. he foowing five abe seection methods are comared: Our roosed agorithm (). Emirica Risk inimization () [4]. seection (). Labe Seection based on Custering () [9]. saming (). When our budget is to seect instances to abe, the method custers the data into custers and then randomy seect one exame from each custer. his method minimizes the rediction error bound reated to abe smoothness, and emiricay erforms the best in [9]. We use Sectra Custering [8] to custer the grah nodes. he resuts of and are both averaged over 0 random trias. and are two methods that iterativey query more data to abe according to the cassifier trained by the reviousy abeed data. seects exames that minimize the emirica risk estimated by the current cassifier. he criterion seects the instances whose abes the current cassifier is the most uncertain. Reca that the harmonic Gaussian fied cassifier adots the one-against-a scheme in muti-cass cassification. Suose we have k casses and u unabeed data oints, then the cassifier oututs a u k score matrix, where each row is for an unabeed oint, and each coumn for a cass. he cass with the argest vaue in the i-th row is the redicted cass of the i-th unabeed oint. Let f (v i ) denote the argest score of node v i reated to a certain cass k, and f 2 (v i ) denote the second argest score of v i reated to a different cass k 2. he smaer f (v i ) f 2 (v i ), the more

6 A Variance inimization Criterion to Active Learning on Grahs abe : Cassification accuracy (%) by using 20 and 50 abes on the Isoet data set. 0 casses 5 casses 20 casses 25 casses 26 casses average # of abes Number of abes (a) 0 Casses Number of abes (b) 5 Casses 0. Number of abes (c) 20 Casses 0. Number of abes (d) 25 Casses 0. Number of abes (e) 26 Casses Figure : Cassification accuracy vs. the number of abes used on the Isoet data set uncertain the cassifier is about the abe rediction of v i. herefore, we seect new instances {v i } to abe with the smaest vaues of f (v i ) f 2 (v i ). his strategy is aso comared in [7]. Notice that and use the abe information of the reviousy seected data, whie other active earning methods do not. In order to test them in our scenario that very itte (if not none) abe information is avaiabe during active earning, for and, we randomy choose an initia set of abes for each of them, rank the other nodes according to the score of their abe seection criterion (emirica risk for, f (v i ) f 2 (v i ) for ), and seect the to ranked nodes. he erformance of and are aso averaged over 0 random seections of the initia set of abes. In the foowing, we begin with a descrition of the data rearation. 4. Data Prearation hree rea-word data sets are used in our exeriments. he first one is the Isoet soken etter database. It contains 50 subjects who soke the name of each etter of the ahabet twice. Hence, we have 52 exames from each seaker. he seakers are groued into sets of 30 seakers each, and are referred to as Isoet, Isoet2, Isoet3, Isoet4, and Isoet5. Here we use Isoet which contains 560 data instances of 26 casses (soken etters). Each cass has 60 exames, and each exame is reresented by a 67-dimensiona vector recording the sectra coefficients, contour features, sonorant features, re-sonorant features and ost-sonorant features. he second one is the NIS handwritten digit htt://archive.ics.uci.edu/m/datasets/isole 56

7 ing Ji, Jiawei Han abe 2: Cassification accuracy (%) by using 20 and 50 abes on the NIS data set. 5 casses 6 casses 7 casses 8 casses 9 casses 0 casses average # of abes Number of abes (a) 5 Casses Number of abes (b) 6 Casses Number of abes (c) 7 Casses Number of abes (d) 8 Casses Number of abes (e) 9 Casses 0. Number of abes (f) 0 Casses Figure 2: Cassification accuracy vs. the number of abes used on the NIS data set database 2. his database has a training set of 60,000 images (denoted as set ), and a testing set of 0,000 images (denoted as set 2). We take the first 000 images from set and the first 000 images from set 2 as our exerimenta data. Each cass (digit) contains around 200 images, each of which is of size and therefore reresented by a 784-dimensiona vector. he third data set is a connected co-author grah extracted from the DBLP database 3 on four areas: machine earning, data mining, information retrieva and database, which naturay form four casses. he coauthor grah contains a tota of 7 vertices, each of which reresents an author. he edge between each air of authors is weighted by the number of aers they co-authored. Each cass (research area) contains around 400 authors. 2 htt://yann.ecun.com/exdb/mnist/ 3 htt:// 562 For each of the first two data sets, Isoet and NIS, foowing [9], we buid a 4-nearest neighbor grah among the data oints, and run the active earning agorithms on grahs as we as the harmonic Gaussian fied cassifier. he third data set contains an inherent grah structure. Note that each data instance (author) in the co-author grah does not have a natura feature reresentation, therefore existing feature-based active earning methods cannot be directy aied to it. 4.2 Cassification Resuts For the Isoet and NIS data sets, the exeriments are conducted by choosing different numbers of casses (denoted as k) from the origina data set. For Isoet, k = 0, 5, 20, 25, 26. For each given cass number k(= 0, 5, 20, 25), the erformance scores are comuted by averaging the scores of 0 reeats of different randomy chosen casses. When k = 26, which is the

8 A Variance inimization Criterion to Active Learning on Grahs abe 3: Cassification accuracy (%) by using 20 and 50 abes on the co-author grah. # of abes Number of abes Figure 3: Cassification accuracy vs. abes used on the co-author grah. the number of tota number of casses in Isoet, we reort the erformance scores of using the whoe data set. For each test, we emoy different active earning methods to seect exames to abe and train a harmonic Gaussian cassifier to redict the abes of the rest of the data. Fig. shows the ots of cassification accuracy versus the number of abes used (). For NIS, the number of casses is chosen to be k = 5, 6, 7, 8, 9, 0, and we aso average the cassification accuracy over 0 different random seections of casses excet for k = 0, which corresonds to using the whoe data set. he cassification accuracy versus the number of abes used is otted in Fig. 2. For the co-author grah, since the origina data set ony contains four casses, we directy run exeriments on the whoe data set. We show the erformance comarison in Fig. 3. As can be observed from Fig. to Fig. 3, our roosed agorithm significanty outerforms other active earning criteria on a the three data sets, eseciay when the number of abes is very sma. erforms the second best on the Isoet and NIS data sets when the number of abes is reativey sma. It is interesting to note that on the NIS data set, and erform not very we when the number of abes is sma, and erform much better when more abes are seected, indicating that they rey heaviy on the abe information of the seected data. We further rovide the detaied cassification accuracy by using 20 and 50 abes in abe 3. he ast two coumns of abe and abe 2 record the average cassification accuracy over different numbers of casses. 563 We can see that overa, erforms significanty better than a the other methods, incuding and that use abe information. Comaring with the agorithm that erforms the second best in each case, achieves 27.0% (%), 29.6% (6.2%), 6.4% (6.6%) reative error reduction in the average cassification accuracy using 20 (50) abes on Isoet, NIS and the co-author grah, resectivey. We have aso erformed the two-taied t-tests at 95% significance eve over the exerimenta resuts in abe 3. In a the cases that erforms the best, the -vaues between the resuts of and other agorithms are ess than herefore, the imrovements of our roosed agorithm are statisticay significant. 5 Concusions From the variance minimization ersective, this aer rooses a nove active earning agorithm urey based on the grah structure, without abe information and feature reresentation on the nodes. One key advantage over existing methods is that our method theoreticay minimizes the exected rediction error of a ouar grah-based cassifier in a batch, offine mode. Exeriments vaidate the effectiveness of our aroach comared to existing active earners on grahs. his study is based on the harmonic Gaussian fied cassifier. here are many other effective grah-based cassifiers with different statistica assumtions of the distribution of the grah data. herefore, it is worthwhie to further anayze the variance and exected rediction error of other earning modes to guide the abe seection over grahs. oreover, this aer seects data to abe urey based on the grah structure. In the future, when the feature reresentation of the nodes is aso avaiabe, it wi be interesting to combine the feature-based active earning criterion and the grah-based active earner together to seect data. 6 Acknowedgements he work was suorted in art by U.S. Nationa Science Foundation grants IIS , IIS-07362, and the U.S. Army Research Laboratory under Cooerative Agreement No. W9NF (NS-CA). he views and concusions contained in this document are those of the authors and shoud not be interreted as reresenting the officia oicies, either exressed or imied, of the Army Research Laboratory or the U.S. Government. he U.S. Government is authorized to reroduce and distribute rerints for Government uroses notwithstanding any coyright notation here on.

9 ing Ji, Jiawei Han References [] Y. Bengio, O. Deaeau, and N. Le Roux. Labe roagation and quadratic criterion. In Semi- Suervised Learning, ages I Press, [2]. Bigic, L. ihakova, and L. Getoor. Active earning for networked data. In ICL, ages 79 86, 200. [3] A. Bum and S. Chawa. Learning from abeed and unabeed data using grah mincuts. In ICL, ages 9 26, 200. [4] A. Bum and. itche. Combining abeed and unabeed data with co-training. In Proceedings of the th Annua Conference on Comutationa Learning heory, ages 92 00, 998. [5] O. Chaee, B. Schökof, and A. Zien, editors. Semi-Suervised Learning. I Press, Cambridge, A, [6] F. R. K. Chung. Sectra Grah heory, voume 92 of Regiona Conference Series in athematics. AS, 997. [7] D. A. Cohn, Z. Ghahramani, and. I. Jordan. Active earning with statistica modes. Journa of Artificia Inteigence Research, 4:29 45, 996. [8] G. H. Goub and C. F. V. Loan. atrix comutations. Johns Hokins University Press, 3rd edition, 996. [9] A. Guiory and J. A. Bimes. Labe seection on grahs. In NIPS, ages , [0] S. Hanneke. An anaysis of grah cut size for transductive earning. In ICL, ages , [] D. A. Harvie. atrix Agebra from a Statistician s Persective. Sringer-Verag, 997. [2] X. He. Laacian Reguarized D-Otima Design for Active Learning and Its Aication to Image Retrieva. IEEE ransactions on Image Processing, 9(): , 200. [3] X. He,. Ji, C. Zhang, and H. Bao. A variance minimization criterion to feature seection using aacian reguarization. IEEE ransactions on Pattern Anaysis and achine Inteigence, 33(0): , 20. [4] S.-J. Huang, R. Jin, and Z.-H. Zhou. Active earning by querying informative and reresentative exames. In NIPS, ages , 200. [5] A. Kuwadekar and J. Nevie. Combining semisuervised earning and reationa resaming for active earning in network domains. In the Budgeted Learning Worksho, ICL, [6] J. Long, J. Yin, W. Zhao, and E. Zhu. Grahbased active earning based on abe roagation. In DAI, ages 79 90, [7] S. A. acskassy. Using grah-based metrics with emirica risk minimization to seed u active earning on networked data. In KDD, ages , [8] A. Y. Ng,. I. Jordan, and Y. Weiss. On sectra custering: Anaysis and an agorithm. In NIPS, ages , 200. [9] K. Peckmans, J. Shawe-ayor, J. A. K. Suykens, and B. D. oor. argin based transductive grah cuts using inear rogramming. Journa of achine Learning Research - Proceedings rack, 2: , [20] B. Settes. Active earning iterature survey. Comuter Sciences echnica Reort 648, University of Wisconsin adison, 200. [2] L. Shi, Y. Zhao, and J. ang. Combining ink and content for coective active earning. In CIK, ages , 200. [22] Z.-J. Zha,. Wang, Y.-. Zheng, Y. Yang, R. Hong, and.-s. Chua. Interactive video indexing with statistica active earning. IEEE ransactions on utimedia, 4():7 27, 202. [23] W. Zhao, J. Long, E. Zhu, and Y. Liu. A scaabe agorithm for grah-based active earning. In Frontiers in Agorithmics, ages 3 322, [24] X. Zhu, Z. Ghahramani, and J. D. Lafferty. Semisuervised earning using gaussian fieds and harmonic functions. In ICL, ages 92 99, [25] X. Zhu, J. Lafferty, and Z. Ghahramani. Combining active earning and semi-suervised earning using gaussian fieds and harmonic functions. In the worksho on he Continuum from Labeed to Unabeed Data in achine Learning and Data ining, ICL, ages 58 65, 2003.