Sparse Greedy Minimax Probability Machine Classification

Transcription

1 Sprse Greed Minim Probbilit Mchine Clssifiction Thoms R. Strohmnn Deprtment of Computer Science Universit of Colordo, Boulder Gregor Z. Grudic Deprtment of Computer Science Universit of Colordo, Boulder Andrei Belitski Deprtment of Computer Science Universit of Colordo, Boulder Dennis DeCoste Mchine Lerning Sstems Group NASA Jet Propulsion Lbortor Abstrct The Minim Probbilit Mchine Clssifiction (MPMC) frmework [Lnckriet et l., 2002] builds clssifiers b minimizing the mimum probbilit of misclssifiction, nd gives direct estimtes of the probbilistic ccurc bound Ω. The onl ssumptions tht MPMC mkes is tht good estimtes of mens nd covrince mtries of the clsses eist. However, s with Support Vector Mchines, MPMC is computtionll epensive nd requires etensive cross vlidtion eperiments to choose kernels nd kernel prmeters tht give good performnce. In this pper we ddress the computtionl cost of MPMC b proposing n lgorithm tht constructs nonliner sprse MPMC (SMPMC) models b incrementll dding bsis functions (i.e. kernels) one t time greedil selecting the net one tht mimizes the ccurc bound Ω. SMPMC utomticll chooses both kernel prmeters nd feture weights without using computtionll epensive cross vlidtion. Therefore the SMPMC lgorithm simultneousl ddresses the problem of kernel selection nd feture selection (i.e. feture weighting), bsed solel on mimizing the ccurc bound Ω. Eperimentl results indicte tht we cn obtin relible bounds Ω, s well s test-set ccurcies tht re comprble to stte of the rt clssifiction lgorithms. 1 Introduction The gol of binr clssifier is to mimize the probbilit tht unseen test dt will be clssified correctl. Assuming tht the test dt is generted from the sme probbilit distribution s the trining dt, it is possible to derive specific probbilit bounds for the cse tht the decision boundr is hperplne. The following result due to Mrshll nd Olkin [1] nd etended b Bertsims nd Popescu [2] provides the theoreticl bsis for

2 ssigning probbilit bounds to hperplne clssifiers: sup P r{ T 1 z b} = E[z]= z,cov[z]=σ z 1 + ω 2 ω 2 = inf T t b(t z) T Σ 1 z (t z) where R d, b re the hperplne prmeters, z is rndom vector, nd t is n ordinr vector. Lnckriet et l (see [3] nd [4]) used the bove results to build the Minim Probbilit Mchine for binr clssifiction (MPMC). From we note tht the onl required relevnt informtion of the underling probbilit distribution for ech clss is it s men nd covrince mtri. No other estimtes nd/or ssumptions re needed, which implies tht the obtined bound (which we refer to s Ω) is essentill distribution-free, i.e. it holds for n distribution with certin men nd covrince mtri. As with other clssifiction lgorithms such s Support Vector Mchines (SVM) (see [5]), the min disdvntge of current MPMC implementtions is tht the re computtionll epensive (sme compleit s SVM), nd require etensive cross vlidtion eperiments to choose kernels nd kernel prmeter to give good performnce on ech dtset. The gol of this pper is to propose kernel bsed MPMC lgorithm tht directl ddresses these computtionl issues. Towrds this end, we propose sprse greed MPMC (SMPMC) lgorithm tht efficientl builds clssifiers, while t the sme time mintins the distribution free probbilit bound of MPM tpe lgorithms. To chieve this gol, we propose to use n itertive lgorithm which dds bsis functions (i.e. kernels) one b one, to n initill empt model. We re considering bsis functions tht re induced b Mercer kernels, i.e. functions of the following form Φ i (z) = K i (z, z i ) (where z i is n input vector of the trining dt). Bses re dded in greed w: we select the prticulr z i tht mimizes the MPMC objective Ω. Furthermore, SMPMC chooses optiml kernel prmeters tht mimize this metric (hence the subscript i in K i ), including utomticll weighting input fetures b γ j 0 for ech kernel dded, such tht z i = (γ 1 z 1, γ 2 z 2,..., γ d z d ) for d dimensionl dt. The proposed SMPMC lgorithm utomticll selects kernels nd re-weights fetures (i.e. does feture selection) for ech new dded bsis function, b minimizing the error bound (i.e. mimizing Ω). Thus the lrge computtionl cost of cross vlidtion (tpicll used b SVM nd MPMC) is voided. The pper is orgnized s follows: Section 2.1 reviews the stndrd MPMC; Section 2.2 describes the proposed sprse greed MPMC lgorithm (SMPMC); nd Sections show how we cn use sprse MPMC to determine optiml kernel prmeters. In section 3 we compre our results to the ones described in the originl MPMC pper (see [4]), showing the probbilit bounds nd the test set ccurcies for different binr clssifiction problems. The conclusion is presented in Section 4. Mtlb source code for the SMPMC lgorithm is vilble online: strohmn/ppers.html 2 Clssifiction model In this section we develop sprse version of the Minim Probbilit Mchine for binr clssifiction. We show tht besides significnt reduction in computtionl cost, the sprse MPMC lgorithm llows us to do utomted kernel nd feture selection. 2.1 Minim Probbilit Mchine for binr clssifiction We will briefl describe the underling concepts of the MPMC frmework s developed in (see [4]). The gol of MPMC is to find decision boundr H(, b) = {z T z = b} such tht the minimum probbilit Ω H of clssifing future dt correctl is mimized. If we ssume tht the two clsses re generted from rndom vectors nd, we cn epress

3 this probbilit bound just in terms of the mens nd covrinces of these rndom vectors: Ω H = inf P (,Σ r{t b T b} (2) ), (ȳ,σ ) Note tht we do not mke n distributionl ssumptions other thn tht, Σ, ȳ, nd Σ re bounded. Eploiting theorem from Mrshll nd Olkin [1], it is possible to rewrite (2) s closed form epression: 1 Ω H = 1 + m 2 (3) where m = min T Σ + T Σ s.t. T ( ȳ) = 1 (4) The optiml hperplne prmeter is the vector tht minimizes (4). The hperplne prmeter b cn then be computed s: b = T T Σ (5) m A new dt point z new is clssified ccording to sign( T z new b ); if this ields +1, z new is clssified s belonging to clss, otherwise it is clssified s belonging to clss. 2.2 Sprse MPM clssifiction One of the ppeling properties of Support Vector Mchines is tht their models tpicll rel onl on smll frction of the trining emples, the so clled support vectors. The models obtined from the kernelized MPMC (see [4]), however, use ll of the trining emples, i.e. the decision hperplne will look like: where in generl ll () i N () N i K( i, z) +, () i 0. () i K( i, z) = b (6) This brings up the question whether one cn lso construct sprse models for the MPMC where most of the coefficients () i or () i re zero. In this pper we propose to do this b strting with n initill empt model nd then dding bsis vectors one b one. As we will see shortl, this pproch is speeding up both lerning nd evlution time while it is still mintining the distribution free probbilit bounds of the MPMC. Before we outline the lgorithm we introduce some nottion: N = N + N the totl number of trining emples l = (l 1,..., l N ) T { 1, 1} N the lbels of the trining dt l (k) = ( l (k) (k) 1,..., l N )T R N output of the model fter dding the kth bsis function (k) = the MPMC coefficients when dding the kth bsis function b (k) = the MPMC offset when dding the kth bsis function Φ b = (K b (b, 1 ),..., K b (b, N ), K b (b, 1 ),..., K b (b, N )) T bsis function evluted on ll trining emples Φ b = (K b (b, 1 ),..., K b (b, N )) T evluted onl on positive emples Φ b = (K b (b, 1 ),..., K b (b, N )) T evluted onl on negtive emples Note tht l (k) is vector of rel numbers (the distnces of the trining dt to the hperplne before ppling the sign function). b R d is the trining emple generting the bsis function Φ b. We will simpl write Φ (k), Φ (k), Φ (k) for the kth bsis. For the first bsis we re solving the one dimensionl MPMC: m = min σ 2 + σ 2 s.t. (Φ Φ Φ Φ ) = 1 (7)

4 Becuse of the constrint the fesible region contins just one vlue for : = 1/(Φ Φ ) b = Φ σ 2 Φ σ 2 Φ + σ 2 Φ = Φ σ Φ σ Φ +σ Φ The first model evluted on the trining dt looks like: l = Φ b (9) All of the subsequent models use the previous estimtion l (k) s one input nd the net bsis Φ (k+1) s the other input. More formll, we set up the two dimensionl clssifiction problem: (k+1) = [ l (k) (k+1) = [ l (k), Φ (k+1), Φ (k+1) ] R N 2 (8) ] R N 2 (10) And solve the following optimiztion problem: m = min T Σ (k+1) + T Σ (k+1) s.t. T ( (k+1) (k+1) ) = 1 (11) Let = ( 1, 2 ) T be the optiml solution of (11). We set: T Σ b (k+1) (k+1) = T (k+1) (12) T Σ (k+1) + T Σ (k+1) nd obtin the net model s: l (k+1) = 1 l (k) + 2 Φ (k+1) b (k+1) (13) As stted bove, one computtionl dvntge of sprse MPMC is tht we tpicll use onl smll number of of trining emples to obtin our finl model (i.e. k << N). Another benefit is tht we hve to solve onl one nd two dimensionl MPMC problems. As seen in (8) the one dimensionl solution is trivil to compute. An nlsis of the two dimensionl problem shows tht it cn be reduced to the problem of finding the roots of fourth order polnomil. Polnomils of degree 4 still hve closed form solutions (see e.g. [6]) which cn be computed efficientl. In the stndrd MPMC lgorithm (see [4]), however, the solution for eqution (4) hs N dimensions nd cn therefore onl be found b epensive numericl methods. It m seem tht the vlues of Ω = 1/(1 + m 2 ) which we obtin from (11) re not true for the whole model since we re considering onl two dimensionl problems nd not ll of the k + 1 dimensions we hve dded so fr through our bsis functions. But it turns out tht the locl bound (from the 2D MPMC) is indeed equl to the globl bound (when considering ll k + 1 dimensions). We stte this fct more formll in the following theorem: Theorem 1: Let l (k) = c 0 + c 1 Φ c k Φ (k) be the sprse MPMC model t the kth itertion (k 1) nd let 1, 2, b (k+1) be the solution of the two dimensionl MPMR: l (k+1) = 1 l (k) + 2 Φ (k+1) b (k+1). Then the vlues of Ω for the two dimensionl MPMC nd for the k + 1 dimensionl MPMC re the sme. Proof: see Appendi 2.3 Selection of bses nd Gussin Kernel widths In our eperiments we re using the Gussin kernel which looks like: K σ (u, v) = ep( u v 2 2 2σ 2 ) (14)

5 Tble 1: Bound Ω, Test set ccurc (TSA), nd number of bses (K) for sprse nd stndrd MPMC Dtset Sprse MPMC Stndrd MPMC (Lnckriet et l.) Ω TSA K Ω TSA K Twonorm 86.4 ± 0.1% 98.3 ± 0.4% ± 0.1% 95.7 ± 0.5% 270 Brest Cncer 90.9 ± 0.1% 96.8 ± 0.3% ± 0.1% 96.9 ± 0.3% 614 Ionosphere 77.7 ± 0.2% 91.6 ± 0.5% ± 0.2% 91.5 ± 0.7% 315 Pim Dibetes 38.2 ± 0.1% 75.4 ± 0.7% ± 0.2% 76.2 ± 0.6% 691 Sonr 78.5 ± 0.2% 86.4 ± 1.0% ± 0.1% 87.5 ± 0.9% 187 where σ is the so clled kernel width. As mentioned before one tpicll hs to choose σ mnull or determine it b tenfold cross vlidtion (see [4]). The sprse MPMC lgorithm greedil selects bsis function out of rndoml chosen cndidte set to mimize Ω which is equivlent to minimizing the vle of m in (7) nd (11). Before we stte the optimiztion problem for the one nd two dimensionl MPMC we rewrite (14) so tht we cn get rid of the denomintor: K γ (u, v) = ep( γ u v 2 2) γ 0 (15) The optimiztion problem we solve for the first itertion is then: min γ m(γ) = min σ 2 Φ + σ 2 s.t. (Φ Φ Φ ) = 1 (16) note tht even though we did not stte it eplicitl the vribles, σ 2 Φ nd Φ ll depend on the kernel prmeter γ., σ 2, Φ Φ, The two dimensionl problem tht hs to be solved for ll subsequent itertions k 2 turns into the following optimiztion problem for γ: min m(γ) = min T Σ (k+1)+ T Σ (k+1) s.t. T ( (k+1) (k+1) ) = 1 (17) γ 2.4 Feture selection For doing feture selection with Gussin kernel one hs to replce the uniform kernel width γ with d dimensionl vector γ of kernel widths: K γ (u, v) = ep( d l=1 γ l(u l v l ) 2 ) (γ l 0 l = 1,..., d) (18) Note tht now the optimiztion problems (16) nd (17) for the one respectivel two dimensionl MPMC re d dimensionl insted of just one dimensionl. 3 Eperiments In this section we describe the results we obtined for sprse MPMC on vrious clssifiction benchmrks. We used the sme dt sets s Lnckriet et l. in [4] for the stndrd MPMC. The dt sets were rndoml divided into 90% trining dt nd 10% test dt nd the results were verged over 50 runs for ech of the five problems (see Tble 3). In ll the eperiments listed in Tble 3 we used the feture selection lgorithm nd hd cndidte set of size 5, i.e. t ech itertion the best bsis out of 5 rndoml chosen cndidtes ws selected. The reults s mesured in test set ccurc re comprble to the ones reported b Lnckriet et l. However, sprse MPMC lws uses in ll cses significntl less bsis functions while still obtining good models. Note lso, tht for the Sonr dt set the probbilit bound Ω of sprse MPMC holds while the one obtined from Stndrd MPMC is invlid. The two plots in figure 1 show wht tpicl lerning curves for sprse MPMC look like. As the number of bsis function increses, both the bound Ω nd the test set ccurc strt to go up nd fter while stbilize. The stbiliztion point usull occurs erlier when one does full feture selection ( γ weight for ech input dimension) insted

6 1 Ionosphere 0.9 Sonr Ω for WS TSA for WS Ω for FS TSA for FS Ω for WS TSA for WS Ω for FS TSA for FS bsis functions bsis functions Figure 1: Bound Ω nd Test Set ccurc (TSA) for width selection (WS) nd feture selection (FS) Test set ccurc Ω bound cndidte 5 cndidtes 10 cndidtes cndidte 5 cndidtes 10 cndidtes bsis functions bsis functions Figure 2: Accurc nd bound for the Dibetes dt set using 1,5 or 10 bsis cndidtes per itertion of kernel width selection (one uniform γ for ll dimensions). We lso eperimented with different sizes for the cndidte set. The plots in figure 2 show wht hppens for 1,5, nd 10 cndidtes. The overll behvior is tht the test set ccurc s well s the Ω vlue converge erlier for lrger cndidte sets (but note tht lrger cndidte set lso increses the computtionl cost per itertion). As seen in figure 1 feture selection gives usull better results in terms of the bound Ω nd the test set ccurc. Furthermore, feture selection lgorithm should indicte which fetures re relevnt nd which re not. We set up n eperiment for the Twonorm dt (which hs 20 input fetures) where we dded 20 dditionl nois fetures tht were not relted to the output. The results re shown in figure 3 nd demonstrte tht the feture selection lgorithm obtined from sprse MPMC is ble to distinguish between relevnt nd irrelevnt fetures. 4 Conclusion & future work This pper introduces new lgorithm (Sprse Minim Probbilit Mchine Clssifiction - SMPMC) for building sprse clssifiction models tht provide lower bound on the probbilit of clssifing future dt correctl. We hve shown tht the method of itertivel dding bsis function hs significnt computtionll dvntges over the stndrd MPMC, while it still mintins the distribution free probbilit bound Ω. Eperimentl

7 weight γ i feture i Figure 3: Averge feture weighting for the Twonorm dt set. The first 20 fetures re the originl inputs, the lst 20 fetures re dditionl nois inputs results indicte tht utomted selection of kernel prmeters, s well s utomted feture selection (weighting), both ke chrcteristics of the SMPMC lgorithm, result in error rtes tht re competitive with those obtined b models where these prmeters must be tuned b computtionll epensive cross-vlidtion. Future reserch on sprse MPMC will focus on estblishing theoreticl frmework for stopping criteri, when dding more bsis functions (kernels) will not significntl reduce error rtes, nd m led to overfitting. Also, eperiments hve so fr focused on using Gussin kernels s bsis functions. From the eperience with other kernel lgorithms, it is known tht other tpe of kernels (polnomil, tnh) cn ield better results for certin pplictions. Furthermore, our frmework is not limited to Mercer kernels, nd other tpes of bsis functions re lso worth investigting. To eperiment with these vritions we provide the Mtlb source code of the lgorithm online: strohmn/ppers.html References [1] A. W. Mrshll nd I. Olkin. Multivrite chebshev inequlities. Annls of Mthemticl Sttistics, 31(4): , [2] I. Popescu nd D. Bertsims. Optiml inequlities in probbilit theor: A conve optimiztion pproch. Technicl Report TM62, INSEAD, Dept. Mth. O.R., Cmbridge, Mss, [3] G. R. G. Lnckriet, L. E. Ghoui, C. Bhttchr, nd M. I. Jordn. Minim probbilit mchine. In T. G. Dietterich, S. Becker, nd Z. Ghhrmni, editors, Advnces in Neurl Informtion Processing Sstems 14, Cmbridge, MA, MIT Press. [4] G. R. G. Lnckriet, L. E. Ghoui, C. Bhttchr, nd M. I. Jordn. A robust minim pproch to clssifiction. Journl of Mchine Lerning Reserch, 3: , [5] B. Schölkopf nd A. Smol. Lerning with Kernels. MIT Press, Cmbridge, MA, [6] Willim H. Beer. CRC Stndrd Mthemthicl Tbles, pge 12. CRC Press Inc., Boc Rton, FL, Appendi: Proof of Theorem 1 We hve to show tht the vlues of m re equl for the two dimensionl MPMC nd the k + 1 dimensionl MPMC. We will just show the equivlence for the first term T Σ,

8 n nlogue rgumenttion will hold for the second term. For the two dimensionl MPMC we hve the following for the term under the squre root: ( ) ( ) ( ) σ2 l 1 (k) (k) σ l Φ (k+1) σ (k+1) Φ l (k) σ 2 Φ (k+1) = [ 1 ] 2 σ (k) l (k+1) 1 2 (k) σ l Φ (k+1) Note tht we cn rewrite σ = Cov(c 0 + c 1 Φ c k Φ (k), c 0 + c 1 Φ (k) 2 l (k) σ l Φ (k+1) = j=1 c ic j Cov(Φ (i), Φ (j) ) = Cov(c 0 + c 1 Φ = c icov(φ (i), Φ (k+1) b using properties of the smple covrince c k Φ (k), Φ (k+1) ) ) + [ 2 ] 2 σ 2 Φ (k+1) c k Φ (k) ) For the k + 1 dimensionl MPMC let us first determine the k + 1 coefficients: l (k+1) = 1 (c 0 + c 1 Φ c k Φ (k) ) + 2 Φ (k+1) b (k+1) = 1 c 1 Φ c k Φ (k) + 2 Φ (k+1) + 1 c 0 b (k+1) The term under the squre root then looks like: T 1 c σ 2... σ 1 Φ... Φ σ Φ (k) Φ Φ (k+1) 1 c c k σ (k) Φ Φ σ Φ (k+1) Φ... σ 2 Φ (k) σ (k) Φ Φ (k+1) 1 c k... σ (k+1) 2 Φ σ 2 Φ (k) Φ (k+1) 2 If we multipl out the ll the elements tht re in row k nd in column k of the covrince mtri we obtin: j=1 (k+1) = [ 1 ] 2 = [ 1 ] 2 σ (k) 2 l ) 1 c j 1 c i Cov(Φ (i), Φ (j) j=1 c ic j Cov(Φ (i), Φ (j) If we multipl out ll the elements tht re either in row k + 1 nd in column k or in row k nd in column k + 1 we obtin: (k+1) 1 c i Cov(Φ (i), Φ (k+1) ) 2 + j=1 (k+1) 2 Cov(Φ (k+1), Φ (j) ) 1 c j = c icov(φ (i), Φ (k+1) ) = (k) σ l Φ (k+1) Finll, we multipl out the lst element in row k + 1 nd column k + 1 [ 2 ] 2 σ 2 Φ (k+1) As we see, ll of the three terms mtch up ectl with the two dimensionl MPMC. Since this will lso hold for the T Σ term in m, we hve shown tht m (nd therefore Ω) is equl for the two dimensionl nd the k + 1 dimensionl MPMC. )