Two view learning: SVM-2K, Theory and Practice

Transcription

1 Two view earning: SVM-2K, Theory and Practice Jason D.R. Farquhar Hongying Meng David R. Hardoon John Shawe-Tayor Sandor Szedmak Schoo of Eectronics and Computer Science, University of Southampton, Southampton, Engand Abstract Kerne methods make it reativey easy to define compex highdimensiona feature spaces. This raises the question of how we can identify the reevant subspaces for a particuar earning task. When two views of the same phenomenon are avaiabe kerne Canonica Correation Anaysis (KCCA) has been shown to be an effective preprocessing step that can improve the performance of cassification agorithms such as the Support Vector Machine (SVM). This paper takes this observation to its ogica concusion and proposes a method that combines this two stage earning (KCCA foowed by SVM) into a singe optimisation termed SVM-2K. We present both experimenta and theoretica anaysis of the approach showing encouraging resuts and insights. 1 Introduction Kerne methods enabe us to work with high dimensiona feature spaces by defining weight vectors impicity as inear combinations of the training exampes. This even makes it practica to earn in infinite dimensiona spaces as for exampe when using the Gaussian kerne. The Gaussian kerne is an extreme exampe, but techniques have been deveoped to define kernes for a range of different datatypes, in many cases characterised by very high dimensionaity. Exampes are the string kernes for text, graph kernes for graphs, margina kernes, kernes for image data, etc. With this pethora of high dimensiona representations it is frequenty hepfu to assist earning agorithms by preprocessing the feature space in projecting the data into a ow dimensiona subspace that contains the reevant information for the earning task. Methods of performing this incude principe components anaysis (PCA) [7, partia east squares [8, kerne independent component anaysis (KICA) [1 and kerne canonica correation anaysis (KCCA) [5.

2 The ast method requires two views of the data both of which contain a of the reevant information for the earning task, but which individuay contain representation specific detais that are different and irreevant. Perhaps the simpest exampe of this situation is a paired document corpus in which we have the same information in two anguages. KCCA attempts to isoate feature space directions that correate between the two views and hence might be expected to represent the common reevant information. Hence, one can view this preprocessing as a denoising of the individua representations through cross-correating them. Experiments have shown how using this as a preprocessing step can improve subsequent anaysis in for exampe cassification experiments using a support vector machine (SVM) [6. This is expained by the fact that the signa to noise ratio has improved in the identified subspace. Though the combination of KCCA and SVM seems effective, there appears no guarantee that the directions identified by KCCA wi be best suited to the cassification task. This paper therefore ooks at the possibiity of combining the two distinct stages of KCCA and SVM into a singe optimisation that wi be termed SVM-2K. The next section introduces the new agorithm and discusses its structure. Experiments are then given showing the performance of the agorithm on an image cassification task. Though the performance is encouraging it is in many ways counter-intuitive, eading to specuation about why an improvement is seen. To investigate this question an anaysis of its generaisation properties is given in the foowing two sections, before drawing concusions. 2 SVM-2K Agorithm We assume that we are given two views of the same data, one expressed through a feature projection φ A with corresponding kerne κ A and the other through a feature projection φ B with kerne κ B. A paired data set is then given by a set S = {(φ A (x 1 ),φ B (x 1 )),...,(φ A (x ),φ B (x ))}, where for exampe φ A coud be the feature vector associated with one anguage and φ B that associated with a second anguage. For a cassification task each data item woud aso incude a abe. The KCCA agorithm ooks for directions in the two feature spaces such that when the training data is projected onto those directions the two vectors (one for each view) of vaues obtained are maximay correated. One can aso characterise these directions as those that minimise the two norm between the two vectors under the constraint that they both have norm 1 [5. We can think of this as constraining the choice of weight vectors in the two spaces. KCCA woud typicay find a sequence of projection directions of dimension anywhere between 50 and 500 that can then be used as the feature space for training an SVM [6. An SVM can be thought of as a 1-dimensiona projection foowed by threshoding, so SVM-2K combines the two steps by introducing the constraint of simiarity between two 1-dimensiona projections identifying two distinct SVMs one in each of the two feature spaces. The extra constraint is chosen sighty differenty from the 2-norm that characterises KCCA. We rather take an ǫ-insensitive 1-norm using sack variabes to measure the amount by which points fai to meet ǫ simiarity: w A,φ A (x i ) + b A w B,φ B (x i ) b B η i + ǫ, where w A, b A (w B, b B ) are the weight and threshod of the first (second) SVM. Combining this constraint with the usua 1-norm SVM constraints and aowing different

3 reguarisation constants gives the foowing optimisation: min L = 1 2 w A w B 2 + C A ξi A + C B such that ξi B + D w A,φ A (x i ) + b A w B,φ B (x i ) b B η i + ǫ y i ( w A,φ A (x i ) + b A ) 1 ξ A i y i ( w B,φ B (x i ) + b B ) 1 ξ B i ξ A i 0, ξ B i 0, η i 0 a for 1 i. Let ŵ A, ŵ B, ˆb A, ˆb B be the soution to this optimisation probem. The fina SVM-2K decision function is then h(x) = sign(f(x)), where f (x) = 0.5 ( ŵ A,φ A (x) + ˆb A + ŵ B,φ B (x) + ˆb ) B = 0.5(f A (x) + f B (x)). Appying the usua Lagrange mutipier techniques we arrive at the foowing dua probem: η i max W = 1 2 ( g A i gj A κ A (x i,x j ) + gi B gj B κ B (x i,x j ) ) + i,j=1 such that gi A = αi A y i β + i + β i, gb i = αi B y i + β + i β i, gi A = 0 = gi B, with the functions 0 α A/B i C A/B 0 β +/ i, β + i + β i D f A/B (x) = 3 Experimenta resuts g A/B i κ A/B (x i,x) + b A/B. (αi A + αi B ) Figure 1: Typica exampe images from the PASCAL VOC chaenge database. Casses are; Bikes (top-eft), Peope (top-right), Cars (bottom-eft) and Motorbikes (bottom-right).

4 The performance of the agorithms deveoped in this paper we evauated on PASCAL Visua Object Casses (VOC) chaenge dataset test1 1. This is a new dataset consisting of four object casses in reaistic scenes. The object casses are, motorbikes (M), bicyces (B), peope (P) and cars (C) with the dataset containing 684 training set images consisting of (214, 114, 84, 272) images in each cass and 689 test set images with (216, 114, 84, 275) for each cass. As can be seen in Figure 1 this is a very chaenging dataset with objects of widey varying type, pose, iumination, occusion, background, etc. The task is to cassify the image according to whether it contains a given object type. We tested the images containing the object (i.e. categories M, B, C and P) against non-object images from the database (i.e. category N). The training set contained 100 positive and 100 negative images. The tests are carried out on 100 new images, haf beonging to the earned cass and haf not. Like many other successfu methods [3, 4 we take a set-of-patches approach to this probem. These methods represent an image in terms of the features of a set of sma image patches. By carefuy choosing the patches and their features this representation can be made argey robust to the common types of image transformation, e.g. scae, rotation, perspective, occusion. Two views were provided of each image through the use of different patch types. One was from affine invariant interest point detectors with a moment invariant descriptor cacuated for each interest point. The second were key point features from SIFT detectors. For one image, severa hundred characteristic patches were detected according to the compexity of the images. These were then custered around K = 400 centres for each feature space. Each image is then represented as a histogram over these centres. So finay, for one image there are two feature vectors of ength 400 that provide the two views. Motorbike Bicyce Peope Car SVM SVM KCCA + SVM SVM 2K Tabe 1: Resuts for 4 datasets showing test accuracy of the individua SVMs and SVM-2K. Figure 1 show the resuts of the test errors obtained for the different categories for the individua SVMs and the SVM-2K. There is a cear improvement in performance of the SVM-2K over the two individua SVMs in a four categories. If we examine the structure of the optimisation, the restriction that the output of the two inear functions be simiar seems to be an arbitrary restriction particuary for points that are far from the margin or are miscassified. Intuitivey it woud appear better to take advantage of the abiities of the different representations to better fit the data. In order to understand this apparent contradiction we now consider a theoretica anaysis of the generaisation of the SVM-2K using the framework provided by Rademacher compexity bounds. 4 Background theory We begin with the definitions required for Rademacher compexity, see for exampe Bartett and Mendeson [2 (see aso [9 for an introductory exposition). Definition 1. For a sampe S = {x 1,,x } generated by a distribution D on a set 1 Avaiabe from VOCdata.tar.gz

5 X and a rea-vaued function cass F with a domain X, the empirica Rademacher compexity of F is the random variabe ˆR (F) = E σ [sup 2 f F σ i f (x i ) x 1,,x where σ = {σ 1,,σ } are independent uniform {±1}-vaued Rademacher random variabes. The Rademacher compexity of F is [ 2 R (F) = E S ˆR (F) = E Sσ [sup σ i f (x i ) We use E D to denote expectation with respect to a distribution D and E S when the distribution is the uniform (empirica) distribution on a sampe S. Theorem 1. Fix δ (0,1) and et F be a cass of functions mapping from S to [0,1. Let (x i ) be drawn independenty according to a probabiity distribution D. Then with probabiity at east 1 δ over random draws of sampes of size, every f F satisfies n(2/δ) E D [f (x) E S [f (x) + R (F) + 3 E S [f (x) + ˆR (F) + 3 f F n(2/δ) Given a training set S the cass of functions that we wi primariy be considering are inear functions with bounded norm { x } α iκ (x i,x) : α Kα B 2 {x w,φ(x) : w B} = F B where φ is the feature mapping corresponding to the kerne κ and K is the corresponding kerne matrix for the sampe S. The foowing resut bounds the Rademacher compexity of inear function casses. Theorem 2. [2 If κ : X X R is a kerne, and S = {x 1,,x } is a sampe of point from X, then the empirica Rademacher compexity of the cass F B satisfies ˆR (F) 2B κ (x i,x i ) = 2B tr (K) 4.1 Anaysing SVM-2K For SVM-2K, the two feature sets from the same objects are (φ A (x i )) and (φ B (x i )) respectivey. We assume the notation and optimisation of SVM-2K given in section 2, equation (1). First observe that an appication of Theorem 1 shows that E S [ f A (x) f B (x) E S [ ŵ A,φ A (x) + ˆb A ŵ B,φ B (x) ˆb B ǫ + 1 η i + 2C n(2/δ) tr(ka ) + tr(k B ) + 3 =: D with probabiity at east 1 δ. We have assumed that w A 2 +b 2 A C2 and w B 2 +b 2 B C 2 for some prefixed C. Hence, the cass of functions we are considering when appying

6 SVM-2K to this probem can be restricted to { ( F C,D = f f : x 0.5 [ g A i κ A (x i,x) + gi B κ B (x i,x) ) + b A + b B, } g A K A g A + b 2 A C 2,g B K B g B + b 2 B C 2, E S [ f A (x) f B (x) D The cass F C,D is ceary cosed under negation. Appying the usua Rademacher techniques for margin bounds on generaisation we obtain the foowing resut. Theorem 3. Fix δ (0,1) and et F C,D be the cass of functions described above. Let (x i ) be drawn independenty according to a probabiity distribution D. Then with probabiity at east 1 δ over random draws of sampes of size, every f F C,D satisfies P (x,y) D (sign(f(x)) y) 0.5 (ξi A + ξi B ) + ˆR n(2/δ) (F C,D ) + 3. It therefore remains to compute the empirica Rademacher compexity of F C,D, which is the critica discriminator between the bounds for the individua SVMs and that of the SVM-2K. 4.2 Empirica Rademacher compexity of F C,D We now define an auxiiary function of two weight vectors w A and w B, D(w A,w B ) := E D [ w A,φ A (x) + b A w B,φ B (x) b B With this notation we can consider computing the Rademacher compexity of the cass F C,D. ˆR (F C,D ) = E σ [ = E σ [ sup 2 sup σ i f (x i ) 1 σ i [ w A,φ A (x i ) + b A + w B,φ B (x i ) + b B f F C,D w A C, w B C D(w A,w B) D Our next observation foows from a reversed version of the basic Rademacher compexity theorem reworked to reverse the roes of the empirica and true expectations: Theorem 4. Fix δ (0,1) and et F be a cass of functions mapping from S to [0,1. Let (x i ) be drawn independenty according to a probabiity distribution D. Then with probabiity at east 1 δ over random draws of sampes of size, every f F satisfies n(2/δ) E S [f (x) E D [f (x) + R (F) + 3 E D [f (x) + ˆR (F) + 3 n(2/δ) The proof tracks that of Theorem 1 but is omitted through ack of space. For weight vectors w A and w B satisfying D(w A,w B ) D, an appication of Theorem 4

7 shows that with probabiity at east 1 δ we have ˆD(w A,w B ) := E S [ w A,φ A (x) + b A w B,φ B (x) b B D + 2C n(2/δ) tr(ka ) + tr(k B ) + 3 ǫ + 1 η i + 4C n(2/δ) tr(ka ) + tr(k B ) + 6 =: ˆD We now return to bounding the Rademacher compexity of F C,D. The above resut shows that with probabiity greater than 1 δ ( ) ˆR FC,D E σ [ sup wa C w B C ˆD(w A,w B) ˆD 1 σ i [ w A,φ A (x i ) + b A + w B,φ B (x i ) + b B First note that the expression in square brackets is concentrated under the uniform distribution of Rademacher variabes. Hence, we can estimate the compexity for a fixed instantiation ˆσ of the the Rademacher variabes σ. We now must find the vaue of w A and w B that maximises the expression 1 [ w A, = 1 ˆσ i φ A (x i ) + b A ˆσ i + w B, ˆσ K A g A + ˆσ K B g B + (b A + b B )ˆσ j subject to the constraints g A K A g A C 2, g B K B g B C 2, and ˆσ i φ B (x i ) + b B ˆσ i 1 1 abs(k A g A K B g B + (b A b B )1) ˆD where 1 is the a ones vector and abs(u) is the vector obtained by appying the abs function to u component-wise. The resuting vaue of the objective function is the estimate of the Rademacher compexity. This is the optimisation soved in the brief experiments described beow. 4.3 Experiments with Rademacher compexity We computed the Rademacher compexity for the probems considered in the experimenta section above. We wished to verify that the Rademacher compexity of the space F C,D, where C and D are determined by appying the SVM-2K, are indeed significanty ower than that obtained for the SVMs in each space individuay. Motorbike Bicyce Peope Car SVM Rad SVM Rad SVM 2K Rad 2K Tabe 2: Resuts for 4 datasets showing test accuracy and Rademacher compexity (Rad) of the individua SVMs and SVM-2K.

8 Tabe 2 shows the resuts for the motorbike, bicyce, peope and car datasets. We show the Rademacher compexities for the individua SVMs and for the SVM-2K aong with the generaisation resuts aready given in Tabe 1. In the case of SVM-2K we samped the Rademacher variabes 10 times and give the corresponding standard deviation. As predicted the Rademacher compexity is significanty smaer for SVM-2K, hence confirming the intuition that ed to the introduction of the approach, namey that the compexity of the cass is reduced by restricting the weight vectors to aign on the training data. Provided both representations contain the necessary data we can therefore expect an improvement in generaisation as observed in the reported experiments. 5 Concusions With the pethora of data now being coected in a wide range of fieds there is frequenty the uxury of having two views of the same phenomenon. The simpest exampe is paired corpora of documents in different anguages, but equay we can think of exampes from bioinformatics, machine vision, etc. Frequenty it is aso reasonabe to assume that both views contain a of the reevant information required for a cassification task. We have demonstrated that in such cases it can be possibe to eaver the correation between the two views to improve cassification accuracy. This has been demonstrated in experiments with a machine vision task. Furthermore, we have undertaken a theoretica anaysis to iuminate the source and extent of the advantage that can be obtained, showing in the cases considered a significant reduction in the Rademacher compexity of the corresponding function casses. References [1 Francis R. Bach and Michae I. Jordan. Kerne independent component anaysis. Journa of Machine Learning Research, 3:1 48, [2 P. L. Bartett and S. Mendeson. Rademacher and Gaussian compexities: risk bounds and structura resuts. Journa of Machine Learning Research, 3: , [3 G. Csurka, C. Bray, C. Dance, and L. Fan. Visua categorization with bags of keypoints. In XRCE Research Reports, XEROX. The 8th European Conference on Computer Vision - ECCV, Prague, [4 R. Fergus, P. Perona, and A. Zisserman. Object cass recognition by unsupervised scae-invariant earning. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, [5 David Hardoon, Sandor Szedmak, and John Shawe-Tayor. Canonica correation anaysis: An overview with appication to earning methods. Neura Computation, 16: , [6 Yaoyong Li and John Shawe-Tayor. Using kcca for japanese-engish cross-anguage information retrieva and cassification. to appear in Journa of Inteigent Information Systems, [7 S. Mika, B. Schökopf, A. Smoa, K.-R. Müer, M. Schoz, and G. Rätsch. Kerne PCA and de-noising in feature spaces. In Advances in Neura Information Processing Systems 11, [8 R. Rosipa and L. J. Trejo. Kerne partia east squares regression in reproducing kerne hibert space. Journa of Machine Learning Research, 2:97 123, [9 J. Shawe-Tayor and N. Cristianini. Kerne Methods for Pattern Anaysis. Cambridge University Press, Cambridge, UK, 2004.