Adaptive Regularization for Transductive Support Vector Machine

Transcription

1 Adaptive Reguarization for Transductive Support Vector Machine Zengin Xu Custer MMCI Saarand Univ. & MPI INF Saarbrucken, Germany Rong Jin Computer Sci. & Eng. Michigan State Univ. East Lansing, MI, U.S. Jianke Zhu Computer Vision Lab ETH Zurich Zurich, Switzerand Irwin King Michae R. Lyu Computer Science & Engineering The Chinese Univ. of Hong Kong Shatin, N.T., Hong Kong Zhirong Yang Information & Computer Science Hesinki Univ. of Technoogy Espoo, Finand Abstract We discuss the framework of Transductive Support Vector Machine (TSVM) from the perspective of the reguarization strength induced by the unabeed data. In this framework, SVM and TSVM can be regarded as a earning machine without reguarization and one with fu reguarization from the unabeed data, respectivey. Therefore, to suppement this framework of the reguarization strength, it is necessary to introduce data-dependant partia reguarization. To this end, we reformuate TSVM into a form with controabe reguarization strength, which incudes SVM and TSVM as specia cases. Furthermore, we introduce a method of adaptive reguarization that is data dependant and is based on the smoothness assumption. Experiments on a set of benchmark data sets indicate the promising resuts of the proposed work compared with state-of-the-art TSVM agorithms. Introduction Semi-supervised earning has attracted a ot of research focus in recenty years. Most of the existing approaches can be roughy divided into two categories: () the custering-based methods [2, 4, 8, 7] assume that most of the data, incuding both the abeed ones and the unabeed ones, shoud be far away from the decision boundary of the target casses; (2) the manifod-based methods make the assumption that most of data ie on a ow-dimensiona manifod in the input space, which incude Labe Propagation [2], Graph Cuts [2], Spectra Kernes [9, 22], Spectra Graph Transducer [], and Manifod Reguarization []. The comprehensive study on semi-supervised earning techniques can be found in the recent surveys [23, 3]. Athough semi-supervised earning wins success in many rea-word appications, there sti remains two major unsoved chaenges. One is whether the unabeed data can hep the cassification, and the other is what is the reation between the custering assumption and the manifod assumption. As for the first chaenge, Singh et a. [6] provided a finite sampe anaysis on the usefuness of unabeed data based on the custer assumption. They show that unabeed data may

2 be usefu for improving the error bounds of supervised earning methods when the margin between different casses satisfies some conditions. However, in the rea-word probems, it is hard to identify the conditions that unabeed data can hep. On the other hand, it is interesting to expore the reation between the ow density assumption and the manifod assumption. Narayanan et a. [4] impied that the cut-size of the graph partition converges to the weighted voume of the boundary which separates the two regions of the domain for a fixed partition. This makes a step forward for exporing the connection between graph-based partitioning and the idea surrounding the ow density assumption. Unfortunatey, this approach cannot be generaized uniformy over a partitions. Lafferty and Wasserman [3] revisited the assumptions of semi-supervised earning from the perspective of imax theory, and suggested that the manifod assumption is stronger than the smoothness assumption for regression. Ti now, the underying reationships between the custer assumption and the manifod assumption are sti undiscosed. Specificay, it is uncear that in what kind of situation the custering assumption or the manifod assumption shoud be adopted. In this paper, we address these current imitations by a unified soution from the perspective of the reguarization strength of the unabeed data. Taking Transductive Support Vector Machine (TSVM) as an exampe, we suggest an framework that introduces the reguarization strength of the unabeed data when estimating the decision boundary. Therefore, we can obtain a spectrum of modes by varying the reguarization strength of unabeed data which corresponds to changing the modes from supervised SVM to Transductive SVM. To seect the optima mode under the proposed framework, we empoy the manifod reguarization assumption that enabes the prediction function to be smooth over the data space. Further, the optima function is a inear combination of supervised modes, weaky semi-supervised modes, and semi-supervised modes. Additionay, it provides an effective approach towards combining the custer assumption and the manifod assumption in semi-supervised earning. The rest of this paper is organized as foows. In Section 2, we review the background of Transductive SVM. In Section 3, we first present a framework of modes with different reguarization strength, foowed by an integrating approach based on manifod reguarization. In Section 4, we report the experimenta resuts on a series of benchmark data sets. Section 5 concudes the paper. 2 Reated Work on TSVM Before presenting the formuation of TSVM, we first describe the notations used in this paper. Let X = (x,...,x n ) denote the entire data set, incuding both the abeed exampes and the unabeed ones. We assume that the first exampes within X are abeed and the next n exampes are unabeed. We denote the unknown abes by y u = (y u +,..., yu n ). TSVM [2] maximizes the margin in the presence of unabeed data and keeps the boundary traversing through ow density regions whie respecting abes in the input space. Under the maximum-margin framework, TSVM aims to find the cassification mode with the maximum cassification margin for both abeed and unabeed exampes, which amounts to sove the foowing optimization probem: w R n,y u R n,ξ R n 2 w K + C ξ i + C i= n i=+ s. t. y i w φ(x i ) ξ i, ξ i 0, i, ξ i () y u i w φ(x i ) ξ i, ξ i 0, + i n, where C and C are the trade-off parameters between the compexity of the function w and the margin errors. Moreover, the prediction function can be formuated as f(x) = w φ(x). Note that we remove the bias term in the above formuation, since it can be taken into account by introducing a constant eement into the input pattern aternativey.

3 As in [9] and [20], we can rewrite () into the foowing optimization probem: f,ξ 2 f K f + C ξ i + C i= n i=+ s. t. y i f i ξ i, ξ i 0, i, f i ξ i, ξ i 0, + i n. ξ i (2) The optimization probem hed in TSVM is a non-inear non-convex optimization [6]. During past severa years, researchers have devoted a significant amount of research efforts to soving this critica probem. A branch-and-bound method [5] was deveoped to search for the optima soution, which is ony imited to sove the probem with a sma number of exampes due to invoving the heavy computationa cost. To appy TSVM for arge-scae probems, Joachims [2] proposed a abe-switching-retraining procedure to speed up the optimization procedure. Later, the hinge oss in TSVM is repaced by a smooth oss function, and a gradient descent method is used to find the decision boundary in a region of ow density [4]. In addition, there are some iterative methods, such as deteristic anneaing [5], concaveconvex procedure (CCCP) [8], and convex reaxation method [9, 8]. Despite the success of TSVM, the unabeed data not necessariy improve cassification accuracy. To better utiize the unabeed data, unike existing TSVM approaches, we propose a framework that tries to contro the reguarization strength of the unabeed data. To do this, we intend to earn the optima reguarization strength configuration from the combination of a spectrum of modes: supervised, weaky-supervised, and semi-supervised. 3 TSVM: A Reguarization View For the sake of iustration, we first study a mode that does not penaize on the cassification errors of unabeed data. Note that the penaization on the margin errors of unabeed data can be incuded if needed. Therefore, we have the foowing form of TSVM that can be derived through the duaity: f,ξ 2 f K f + C ξ i (3) i= s. t. y i f i ξ i, ξ i 0, i, f 2 i, + i n. 3. Fu Reguarization of Unabeed Data In order to adjust the strength of the reguarization raised from the unabeed exampes, we introduce a coefficient ρ 0, and modify the above probem (3) as beow: f,ξ 2 f K f + C ξ i (4) i= s. t. y i f i ξ i, ξ i 0, i, fi 2 ρ, + i n. Obviousy, it is the standard TSVM for ρ =. In particuar, the arger the ρ is, the stronger the reguarization of unabeed data is. It is aso important to note that we ony take into account the cassification errors on the abeed exampes in the above equation. Namey, we ony denote ξ i for each abeed exampe. Further, we write f = (f ;f u ) where f = (f,..., f ) and f u = (f +,...,f n ) represent the prediction for the abeed and the unabeed exampes, respectivey. According to the inverse emma of the bock matrix, we can write K as foows: ( K = M M u K u,k, K, K,uM u M u ),

4 where Thus, the term f K f is computed as M = K, K,u K u,u K u,, M u = K u,u K u, K, K,u. f K f = f M f + fu M u f u 2f K, K,uM u f u. When the unabeed data are oosey correated to the abeed data, namey when most of the eements within K u, are sma, this eads to M u K u. We refer to this case as weaky unsupervised earning. Using the above equations, we rewrite TSVM as foows: f,f u,ξ 2 f M f + C ξ i + ω(f, ρ) (5) i= s. t. y i f i ξ i, ξ i 0, i, where ω(f, ρ) is a reguarization function for f and it is the resut of the foowing optimization probem: f u 2 f u M u f u f K, K,uM u f u (6) s. t. [fi u ] 2 ρ, + i n. To understand the reguarization function ω(f, ρ), we first compute the dua of the probem (6) by the Lagrangian function: L = 2 f u M u f u f K, K,uM u f u = 2 f u (M u n u i= 2 λ i([f u i ]2 ρ) D(λ))f u f K, K,uM u f u + ρ 2 λ e, where D(λ) = diag(λ,...,λ n ) and e denotes a vector with a eements being one. As the derivatives vanish for optimaity, we have where I is an identity matrix. f u = (M u D(λ)) M u K u,k, f = (I M u D(λ)) K u, K, f, Repacing f u in (6) with the above equation, we have the foowing dua probem: max λ 2 f K, K,u(M u M u D(λ)M u ) K u, K, f + ρλ e (7) s. t. M u D(λ), λ i 0, i =,...,n. The above formuation aows us to understand how the parameter ρ contros the strength of reguarization from the unabeed data. In the foowing, we wi show that a series of earning modes can be derived through assigning various vaues for the coefficient ρ. 3.2 No Reguarization from Unabeed Data First, we study the case of ρ = 0. We have the foowing theorem to iustrate the reationship between the dua probem (7) and the supervised SVM. Theorem When ρ = 0, the optimization probem is reduced to the standard supervised SVM.

5 Proof It is not difficut to see that the optima soution to (7) is λ = 0. As a resut, ω(f, ρ) becomes ω(f, ρ = 0) = 2 f K, K,uM u K u, K, f Substituting ω(f, ρ) in (5) with the formuation above, the overa optimization probem becomes f,ξ 2 f (M K, K,uM u K u, K, )f + C ξ i s. t. y i f i ξ i, ξ i 0, i. According to the matrix inverse emma, we cacuate M M = (K, K,u K u,u K u,) as beow: i= = K, + K, K,u(K u,u K u, K, K,u) K u, K, = K, + K, K,uM u K u, K,. Finay, the optimization probem is simpified as f,ξ 2 f K, f + C ξ i (8) i= s. t. y i f i ξ i, ξ i 0, i. Ceary, the above optimization is identica to the standard supervised SVM. Hence, the unabeed data are not empoyed to reguarize the decision boundary when ρ = Partia Reguarization of Unabeed Data Second, we consider the case when ρ is sma. According to (7), we expect λ to be sma when ρ is sma. As a resut, we can approximate (M u M u D(λ)M u ) as foows: (M u M u D(λ)M u ) M u Consequenty, we can write ω(f, ρ) as foows: + D(λ). ω(f, ρ) = 2 f K, K,uM u K u,k, f + φ(f, ρ), (9) where φ(f, ρ) is the output of the foowing optimization probem max λ ρλ e 2 f K, K,uD(λ)K u, K, f s. t. M u D(λ), λ i 0, i =,...,n. We can simpify the above probem by approximating M u D(λ) as λ i [σ (M u )], i =,...,n, where σ (M u ) represents the maximum eigenvaue of matrix M u. The resuting simpified probem becomes ρ max λ 2 λ e 2 f K, K,uD(λ)K u, K s. t. 0 λ i [σ (M u )], i n. As the above probem is a inear programg probem, the soution for λ can be computed as: { 0 [K u, K, λ i = f ] 2 i > ρ, σ(m u ) [K u, K, f ] 2 i ρ. From the above formuation, we find that ρ pays the roe of a threshod of seecting the unabeed exampes. Since [K u, K, f ] i can be regarded as the approximation for the ith, f

6 unabeed exampe, the above formuation can be interpreted in the way that ony the unabeed exampes with ow prediction confidence wi be seected for reguarizing the decision boundary. Moreover, a the unabeed exampes with high prediction confidence wi be ignored. From the above discussions, we can concude that ρ deteres the reguarization strength of unabeed exampes. Then, we rewrite the overa optimization probem as beow: max f,ξ λ 2 f K, f + C i= s. t. y i f i ξ i, ξ i 0, i, 0 λ i [σ (M u )], i n. ξ i 2 f K, K,uD(λ)K u, K, f (0) This is a -max optimization probem and thus the goba optima soution can be guaranteed. To obtain the optima soution, we empoy an aternating optimization procedure, which iterativey computes the vaues of f and λ. To account for the penaty on the margin error from the unabeed data, we just need to add an extra constraint of λ i 2C for i =,...,n. By varying the parameter ρ from 0 to, we can indeed obtain a series of transductive modes for SVM. When ρ is sma, we ca the corresponding optimization probem as weaky semisupervised earning. Therefore, it is important to find an appropriate ρ which adapts for the input data. However, as the data distribution is usuay unknown, it is very chaenging to directy estimate an optima reguarization strength parameter ρ. Instead, we try to expore an aternative approach to seect an appropriate ρ by combining the prediction functions. Due to the arge cost in cacuating the inverse of kerne matrices, one can sove the dua probems according to the Representer theorem. 3.4 Adaptive Reguarization As stated in previous sections, ρ deteres the reguarization strength of the unabeed data. We now try to adapt the parameter ρ according to the unabeed data information. Specificay, we intend to impicity seect the best ρ from a given ist, i.e., Υ = {ρ,..., ρ m } where ρ = 0 and ρ m =. This is equivaent to seecting the optima f from a ist of prediction functions, i.e., F = {f,...,f m }. Motivated from the ensembe technique for semi-supervised earning [7], we assume that the optima f comes from a inear combination of the base functions {f i }. We then have: f = m θ i f i, i= m θ i =, θ i 0, i =,...,m. i= where θ i is the weight of the prediction function f i and θ R m. One can aso invove a priori to θ i. For exampe, if we have more confidences on the semi-supervised cassifier, we can introduce a constraint ike θ m 0.5. It is important to note that the earning functions in ensembe methods [7] are usuay weak earners, whie in our approach, the earning functions are strong earners with different degrees of reguarization. In the foowing, we study how to set the reguarization strength adaptive to data. Since TSVM naturay foows the custer assumption of semi-supervised earning, in order to compement the custer assumption, we adopt another principe in semi-supervised earning, i.e., the manifod assumption. From the point of view of manifod assumption in semisupervised earning, the prediction function f shoud be smooth on unabeed data. To this end, the approach of manifod reguarization is widey adopted as a smoothing term in semi-supervised earning iteratures, e.g., [, 0]. In the foowing, we wi empoy the manifod reguarization principe for seecting the reguarization strength. The manifod reguarization is mainy based on a graph G =< V, E > derived from the whoe data space X, where V = {x i } n i= is the vertex set, and E denotes the edges inking pairs of nodes. In genera, a graph is buit in the foowing four steps: () constructing adjacency graph; (2) cacuating the weights on edges; (3) computing the adjacency matrix W; (4)

7 obtaining the graph Lapacian by L = diag( n j= W ij) W. Then, we denote the manifod reguarization term as f Lf. For simpicity, we denote the predicted vaues of function f i on the data X as f i, such that f i = ([f i ],..., [f i ] n ). F = (f,...,f m ) is used to represent the set of the prediction vaues of a prediction functions. Finay, We have the foowing imization probem: θ 2 η(θ F)L(F θ) y (F θ) () s. t. θ e =, θ i 0, i =,...,m, where the second term, y (F θ), is used to strengthen the confidence on the prediction over the abeed data. η is a trade-off parameter. The above optimization probem is a simpe quadratic programg probem, which can be soved very efficienty. It is important to note that the above optimization probem is ess sensitive to the graph structure than Lapacian SVM as used in [], since the basic earning functions are a strong earners. It aso saves a huge amount of efforts in estimating the parameters compared with Lapacian SVM. The above approach indeed provides a practica approach towards a combination of both the custer assumption and the manifod assumption. It is empiricay suggested that combining these two assumptions heps to improve the prediction accuracy of semi-supervised earning according to the survey paper on semi-supervised SVM [6]. Moreover, when ρ = 0, supervised modes are incorporated in the framework. Thus the usefuness of unabeed in naturay considered by the reguarization. This therefore provides a practica soution to the probems described in Section. 4 Experiment In this section, we give detais of our impementation and discuss the resuts on severa benchmark data sets for our proposed approach. To conduct a comprehensive evauation, we empoy severa we-known datasets as the testbed. As summarized in Tabe, three image data sets and five text data sets are seected from the recent book ( mpg.de/ss-book/) and the iterature ( Tabe : Datasets used in our experiments. d represents the data dimensionaity, and n denotes the tota number of exampes. Data set n d Data set n d usps digit coi ibm vs rest pcmac page ink pageink For simpicity, our proposed adaptive reguarization approach is denoted as ARTSVM. To evauate it, we conduct an extensive comparison with severa state-of-the-art approaches, incuding the abe-switching-retraining agorithm in SVM-Light [2], CCCP [8], and TSVM [4]. We empoy SVM as the baseine method. In our experiments, we repeat a the agorithms 20 times for each dataset. In each run, 0% of the data are randomy seected as the training data and the remaining data are used as the unabeed data. The vaue of C in a agorithms are seected from [, 0, 00, 000] using cross-vaidation. The set of ρ is set to [0, 0.0, 0.05, 0., ] and η is fixed to As stated in Section 3.4, ARTSVM is ess sensitive to the graph structure. Thus, we adopt a simpe way to construct the graph: for each data, the number of neighbors is set to 20 and binary weighting is empoyed. In ARTSVM, the supervised, weaky semi-supervised, and semi-supervised agorithms are based on impementation in LibSVM ( tw/~cjin/ibsvm/), MOSEK ( and TSVM ( de/bs/peope/chapee/ds/), respectivey. For the comparison agorithms, we adopt the origina authors own impementations. Tabe 2 summarizes the cassification accuracy and the standard deviations of the proposed ARTSVM method and other competing methods. We can draw severa observations from

8 the resuts. First of a, we can ceary see that our proposed agorithm performs significanty better than the baseine SVM method across a the data sets. Note that some very arge deviations in SVM are mainy because the abeed data and the unabeed data may have quite different distributions after the random samping. On the other hand, the unabeed data capture the underying distribution and hep to correct such random error. Comparing ARTSVM with other TSVM agorithms, we observe that ARTSVM achieves the best performance in most cases. For exampe, for the digita image data sets, especiay digit, supervised earning usuay works we and the advantages of TSVM are very imited. However, the proposed ARTSVM outperforms both the supervised and other semisupervised agorithms. This indicates that the appropriate reguarization from the unabe data improves the cassification performance. Tabe 2: The cassification performance of Transductive SVMs on benchmark data sets. Data Set ARTSVM TSVM SVM CCCP SVM-ight usps 8.30± ± ± ± ±4.4 digit 82.0± ± ± ± ±4.24 coi 8.70± ± ± ± ±2.84 ibm vs rest 78.04± ± ± ± ±5.8 pcmac 95.50± ± ± ± ±7.24 page 94.65± ± ± ± ±2.60 ink 94.27± ± ± ± ±2.45 pageink 97.3± ± ± ± ±.8 5 Concusion This paper presents a nove framework for semi-supervised earning from the perspective of the reguarization strength from the unabeed data. In particuar, for Transductive SVM, we show that SVM and TSVM can be incorporated as specia cases within this framework. In more detai, the oss on the unabeed data can essentiay be regarded as an additiona reguarizer for the decision boundary in TSVM. To contro the reguarization strength, we introduce an aternative method of data-dependant reguarization based on the principe of manifod reguarization. Empirica studies on benchmark data sets demonstrate that the proposed framework is more effective than the previous transductive agorithms and purey supervised methods. For future work, we pan to design a controing strategy that is adaptive to data from the perspective of ow density assumption and manifod reguarization of semi-supervised earning. Finay, it is desirabe to integrate the ow density assumption and manifod reguarization into a unified framework. Acknowedgement The work was supported by the Nationa Science Foundation (IIS ), Nationa Institute of Heath (R0GM ), Research Grants Counci of Hong Kong (CUHK458/08E and CUHK428/08E), and MSRA (FY09-RES-OPP-03). It is aso affiiated with the MS-CUHK Joint Lab for Human-centric Computing & Interface Technoogies. References [] Mikhai Bekin, Partha Niyogi, and Vikas Sindhwani. Manifod reguarization: A geometric framework for earning from abeed and unabeed exampes. Journa of Machine Learning Research, 7: , [2] Avrim Bum and Shuchi Chawa. Learning from abeed and unabeed data using graph cuts. In ICML 0: Proceedings of the 8th internationa conference on Machine earning, pages Morgan Kaufmann, San Francisco, CA, 200. [3] O. Chapee, B. Schökopf, and A. Zien, editors. Semi-Supervised Learning. MIT Press, Cambridge, MA, 2006.

9 [4] O. Chapee and A. Zien. Semi-supervised cassification by ow density separation. In Proceedings of the Tenth Internationa Workshop on Artificia Inteigence and Statistics, pages 57 64, [5] Oivier Chapee, Vikas Sindhwani, and Sathiya Keerthi. Branch and bound for semi-supervised support vector machines. In B. Schökopf, J. Patt, and T. Hoffman, editors, Advances in Neura Information Processing Systems 9. MIT Press, Cambridge, MA, [6] Oivier Chapee, Vikas Sindhwani, and Sathiya S. Keerthi. Optimization techniques for semisupervised support vector machines. Journa of Machine Learning Research, 9: , [7] Ke Chen and Shihai Wang. Reguarized boost for semi-supervised earning. In J.C. Patt, D. Koer, Y. Singer, and S. Roweis, editors, Advances in Neura Information Processing Systems 20, pages MIT Press, Cambridge, MA, [8] Ronan Coobert, Fabian Sinz, Jason Weston, and Léon Bottou. Large scae transductive SVMs. Journa of Machine Learning Reseaerch, 7:687 72, [9] S. C. H. Hoi, M. R. Lyu, and E. Y. Chang. Learning the unified kerne machines for cassification. In Proceedings of Twentith Internationa Conference on Knowedge Discovery and Data Mining (KDD-2006), pages 87 96, New York, NY, USA, ACM Press. [0] Steven C. H. Hoi, Rong Jin, and Michae R. Lyu. Learning nonparametric kerne matrices from pairwise constraints. In ICML 07: Proceedings of the 24th internationa conference on Machine earning, pages , New York, NY, USA, ACM. [] T. Joachims. Transductive earning via spectra graph partitioning. In ICML 03: Proceedings of the 20th internationa conference on Machine earning, pages , [2] Thorsten Joachims. Transductive inference for text cassification using support vector machines. In ICML 99: Proceedings of the 6th internationa conference on Machine earning, pages , San Francisco, CA, USA, 999. Morgan Kaufmann Pubishers Inc. [3] John Lafferty and Larry Wasserman. Statistica anaysis of semi-supervised regression. In J.C. Patt, D. Koer, Y. Singer, and S. Roweis, editors, Advances in Neura Information Processing Systems 20, pages MIT Press, Cambridge, MA, [4] Hariharan Narayanan, Mikhai Bekin, and Partha Niyogi. On the reation between ow density separation, spectra custering and graph cuts. In B. Schökopf, J. Patt, and T. Hoffman, editors, Advances in Neura Information Processing Systems 9, pages MIT Press, Cambridge, MA, [5] Vikas Sindhwani, S. Sathiya Keerthi, and Oivier Chapee. Deteristic anneaing for semisupervised kerne machines. In ICML 06: Proceedings of the 23rd internationa conference on Machine earning, pages , New York, NY, USA, ACM Press. [6] Aarti Singh, Robert Nowak, and Xiaojin Zhu. Unabeed data: Now it heps, now it doesn t. In D. Koer, D. Schuurmans, Y. Bengio, and L. Bottou, editors, Advances in Neura Information Processing Systems 2, pages [7] Junhui Wang, Xiaotong Shen, and Wei Pan. On efficient arge margin semisupervised earning: Method and theory. Journa of Machine Learning Research, 0:79 742, [8] Lini Xu and Dae Schuurmans. Unsupervised and semi-supervised muti-cass support vector machines. In AAAI, pages , [9] Zengin Xu, Rong Jin, Jianke Zhu, Irwin King, and Michae R. Lyu. Efficient convex reaxation for transductive support vector machine. In J.C. Patt, D. Koer, Y. Singer, and S. Roweis, editors, Advances in Neura Information Processing Systems 20, pages MIT Press, Cambridge, MA, [20] T. Zhang and R. Ando. Anaysis of spectra kerne design based semi-supervised earning. In Y. Weiss, B. Schökopf, and J. Patt, editors, Advances in Neura Information Processing Systems (NIPS 8), pages MIT Press, Cambridge, MA, [2] Dengyong Zhou, Oivier Bousquet, Thomas Navin La, Jason Weston, and Bernhard Schökopf. Learning with oca and goba consistency. In Sebastian Thrun, Lawrence Sau, and Bernhard Schökopf, editors, Advances in Neura Information Processing Systems 6. MIT Press, Cambridge, MA, [22] X. Zhu, J. Kandoa, Z. Ghahramani, and J. Lafferty. Nonparametric transforms of graph kernes for semi-supervised earning. In Advances in Neura Information Processing Systems (NIPS 7), pages , Cambridge, MA, MIT Press. [23] Xiaojin Zhu. Semi-supervised earning iterature survey. Technica report, Computer Sciences, University of Wisconsin-Madison, 2005.