Maximum Margin Bayesian Networks

Transcription

1 Mxmum Mrgn Besn Networks Yuhong Guo Deprtment of Computng Scence Unverst of Albert Lnl Xu School of Computer Scence Unverst of Wterloo Dle Schuurmns Deprtment of Computng Scence Unverst of Albert Abstrct We consder the problem of lernng Besn network clssfers tht mxmze the mrgn over set of clssfcton vrbles. We fnd tht ths problem s hrder for Besn networks thn for undrected grphcl models lke mxmum mrgn Mrkov networks, snce the prmeters n Besn network must stsf ddtonl normlzton constrnts tht n undrected grphcl model need not respect. Unfortuntel, these normlzton constrnts destro the convext propertes of the trnng problem nd sgnfcntl complcte the optmzton tsk. Nevertheless, we derve n effectve trnng lgorthm tht solves the mxmum mrgn trnng problem for rnge of network topologes, nd otherwse converges to locll optml set of prmeters for rbtrr network topologes. Expermentl results show tht the method hs promse, lthough the complext of the optmzton poses nontrvl brrer n prctce. Our mn ntent s smpl to pose nd nvestgte wht we beleve s nturl mchne lernng pproch, whle lso pontng out ts dffcultes. 1 INTRODUCTION When trnng probblt models for clssfcton tsks t s often recommended tht the model prmeters be optmzed under dscrmntve trnng crteron such s condtonl lkelhood (Fredmn et l., However, generl Besn network clssfers hve rrel, f ever, been trned to mxmze the mrgn rgubl the most dscrmntve crteron vlble. Recentl t hs been observed tht undrected grphcl models cn be effcentl trned to mxmze the mrgn, even smultneousl over set of clssfcton vrbles (Tskr et l., (An nterestng precursor s (Altun et l., However, followng SVMs, these trnng lgorthms hve dopted the Euclden normlzton constrnt of support vector mchnes, whch cn be ccommodted n ther frmeworks becuse the rel on n undrected grphcl model representton. In ths pper we consder pplng the mxmum mrgn methodolog to Besn networks. Unlke Mrkov network models, Besn networks requre the strong locl normlzton constrnts be stsfed. These constrnts re t odds wth the stndrd Euclden (or L p normlzton constrnts of SVMs. Nevertheless, our gol s to explore the possblt of lernng mxmum mrgn clssfers whle stll beng ble to represent the lerned clssfer s Besn network. There re severl motvtons for ttemptng to mntn Besn network representton. Frst, the clssfcton problem could be frgment of much lrger probblstc cusl model, nd mntnng Besn network representton wll llow one to ntegrte the lerned model wth the rest of the model semlessl. Second, the normlzton constrnts sserted b Besn network structure cpture cusl knowledge bout the domn. Respectng these constrnts s one w to explot the dvntge of Besn networks exhbt for ntutvel modelng the cusl structure of domn. Removng the normlzton constrnts turns the Besn network nto Mrkov network, nd therefore must necessrl lose the orgnl cusl knowledge tht ws encoded n the constrnts. The remnder of the pper s orgnzed s follows. Frst, fter prelmnr defntons, we nvestgte the noton of mrgn for Besn network clssfers n Secton 3, nd relte ths both to the common condtonl lkelhood crteron of grphcl models, nd to the stndrd mrgn defnton of SVMs. We then derve n effectve trnng lgorthm n Secton 4 tht solves wde rnge of problems exctl, nd otherwse

2 provdes n effectve heurstc for fndng locl solutons. In Secton 5 we present expermentl results whch show some evdence tht the cusl nformton n Besn networks cn help mxmum mrgn trnng. Fnll, we extend the pproch to multvrble clssfcton n Secton 6. In the end, we observe few drwbcks of mxmum mrgn Besn networks (ncludng the fct tht the do not llow the kernel trck to be convententl ppled, nd thus the mn messge of ths pper s necessrl mxed: On the one hnd, mxmum mrgn Besn networks llow one to explot cusl pror knowledge effectvel, but on the other hnd the crete ddtonl computtonl dffcult whle blockng the stndrd kernel trck. Nevertheless, mxmum mrgn Besn networks re nturl combnton of two predomnnt current lernng technologes, nd we feel ths combnton s worth stud. 2 BAYESIAN NETWORKS We ssume we re gven Besn network whch s defned b drected cclc grph over vrbles X 1,..., X n where the probblt of complete confgurton x s gven b P (x θ = n P (x j x π(j j=1 ( = exp jb 1 (x j=b ln θ jb (1 Here θ denotes the prmeters of the model, j rnges over CPTs, one for ech vrble X j, 1 ( denotes the ndctor functon, x j denotes the locl subconfgurton of x on (x j, x π(j, denotes the set of vlues for chld vrble x j, nd b denotes the set of confgurtons for x j s prents x π(j. The form (1 shows how Besn networks re form of exponentl model P (x w = exp ( δ(x w (2 usng the substtuton w jb = ln θ jb, where δ(x denotes the feture vector (...1 (xj=b... over j,, b. The ke spect of the exponentl form s tht t expresses p(x w s convex functon of the prmeters w, whch would seem to suggest convenent optmzton problems. However, Besn networks lso requre the mposton of ddtonl normlzton constrnts over ech vrble = 1 for ll j, b (3 Unfortuntel, these constrnts re nonlner, even though the objectve s convex n w. Removng these constrnts mproves the computtonl dffcult of trnng, but lso removes the cusl nterpretblt of the model. In ths pper, our gol s to stck wth the Besn network constrnts nd dscover where ths leds. 3 DISCRIMINATIVE TRAINING We ntll ssume there s sngle clssfcton vrble Y tkng on vlues {1,.., K}. To mke predctons we wll consder the mxmum condtonl probblt predcton mx P ( x. Note tht for grphcl models the condtonl probblt depends onl on vrbles tht shre some common functon (CPT wth Y (the Mrkov blnket of Y, nd therefore we wll restrct ttenton to ths set of vrbles henceforth. We re nterested n lernng the prmeters for Besn network clssfer gven trnng dt of the form (x 1 1,..., (x t t. Two stndrd trnng crter to mxmze durng trnng re the jont loglkelhood nd the condtonl loglkelhood gven b log L(θ = log CLL(θ = t log P ( x θ + log P (x θ(4 =1 t log P ( x θ (5 =1 Much of the lterture suggests tht the ltter objectve s better suted for clssfcton (Lffert et l., 2001; Fredmn et l., 1997, lthough recent studes hve dentfed condtons where the former objectve s dvntgeous (Ng & Jordn, In ths pper we consder two lterntve crter bsed on the lrge mrgn crter of SVMs, whch we refer to s mnmum condtonl lkelhood (MCL nd mnmum condtonl lkelhood rto (MCLR respectvel log M CL(θ = mn log P ( x θ (6 log M CLR(θ = mn log P ( x θ 1 K K log P ( x θ (7 =1 For the two clss cse, K = 2, these two crter re n fct equvlent. Also n ths cse, the re both ver smlr to condtonl loglkelhood (5, dfferng onl n tkng mn nsted of sum cross trnng exmples. Now, b pluggng n the exponentl form of the defnton of P ( xw nto these crter we wll be ble to relte the resultng trnng problem to tht of lner

3 SVMs log M CL(w = mn δ(x w log log M CLR(w = mn e δ(x w (8 K [ δ(x δ(x ] w (9 =1 The gol here s to mxmze these qunttes wth respect to the weght vector w. Of course, mxmzng these nner products s trvl f w s not constrned. At ths pont, the stndrd SVM formulton mposes Euclden normlzton constrnt tht w 2 = 1, whch sets the weghts to mxmze the Euclden mrgn (Schoelkopf & Smol, For the second crteron specfcll, our formulton recovers stndrd versons of multclss SVMs proposed n (Crmmer & Snger, 2001 (gnorng slcks expressed over fetures determned b the Besn network. Ths specfc connecton s the mn observton of (Tskr et l., 2003; Altun et l., 2003, who proceed to use stndrd SVM trnng crter over these fetures. (We consder multvrble clssfcton n Secton 6 below. Note however tht the soluton weght vector for ths problem cnnot be substtuted nto the Besn network representton, becuse t wll not stsf the proper normlzton constrnts (3. The prevous technques of (Tskr et l., 2003; Altun et l., 2003 were ble to proceed b usng n undrected grphcl model whch cn ccomodte unnormlzed weghts n the potentl functon. However, for Besn networks ths s not suffcent, nd there s usull no w to represent the sme clssfer n the orgnl Besn network structure. Our pproch tht we consder n ths pper s to preserve representblt s Besn network, whch requres one to solve the mxmum mrgn trnng crter (8 nd (9 wth respect to the lterntve normlzton constrnts (3. Unfortuntel, the constrnts n w re hghl nonlner nd ths elds dffcult optmzton problem. Attempts to reformulte the problem ccordng to stndrd trnsformtons lso fl. For exmple, the probblt functon (1 s nether concve nor convex n the prmeters θ, even though the eqult constrnts re lner. The stndrd trck to remove the normlzton constrnts entrel lso does not work n ths cse, snce the stndrd reprmeterzton θ jb = e ω jb / eω jb cretes n objectve [ P (x ω = exp 1 (xj=b jb ω jb log e ω jb] tht s nether convex nor concve over ω. Thus, f we hope to solve the mxmum mrgn Besn network trnng problem exctl, even for specl cses, we requre more subtle pproch. 4 A TRAINING ALGORITHM Although solvng for the mxmum mrgn Besn network prmeters ppers to be hrd n generl, we cn derve prctcl trnng lgorthm tht stll solves the problem for wde rnge of grph topologes, nd otherwse provdes useful foundton for heurstc pproches whch seek locl mxm. The mn de s to tr to explot convext n the problem s much s possble, nd dentf stutons where the solutons to convex subproblem cn be mntned. Below we wll work wth the MCL crteron (8 lthough smlr dervton lso works for (9. Note frst tht (8 s convex objectve functon n w. Unfortuntel, we hve to mxmze (8 wth respect to the nonlner eqult constrnts (3. However, the bsc observton s tht the problem cn be mde convex smpl b relxng these eqult constrnts to neqult constrnts, nd thus obtn smple relxton of the problem whch llows us to obtn globl soluton rg mx mn w = rg mn w,β δ(x w log subject to β subject to e δ(x w 1 for ll j, b e δ(x w (10 β δ(x w + log for ll j, b (11 The soluton to ths problem wll of course be subnormlzed. The ke fct bout the relxed optmzton problem (11 however, s tht t s convex n w nd ths wll permt effectve lgorthmc pproches. For ths problem we cn obtn the Lgrngn L 0 (w, β, µ, λ = β + ( µ β δ(x w + log + j,b λ jb ( 1 Ths gves us n equvlent problem to (11 e δ(x w mn mx L 0(w, β, µ, λ subject to µ 0, λ 0 (12 w,β µ,λ Frst, t turns out to be es to elmnte β from ths problem, snce L 0 / β = 1 + µ, nd settng ths

4 to 0 mples µ = 1. If we enforce ths constrnt, we cn plug ths equton bck nto the Lgrngn L 1 (w, µ, λ = ( µ log e δ(x w δ(x w + j,b λ jb ( 1 An equvlent optmzton problem to (12 s therefore mn mx L 1(w, µ, λ s.t. µ 0, w µ,λ µ = 1, λ 0 (13 Becuse µ nd λ re nonnegtve, L 1 s convex n w nd lner n µ nd λ, nd therefore ths problem onl hs globl solutons. We now ttempt to solve for w for gven µ nd λ. Tkng the prtl dervtve wth repect to w jb nd settng ths equl to 0 shows tht we seek w tht stsfes µ p( jb x + λ jb = µ δ jb (,(14 x for ll j,, b, where here we hve used the substtuton dvded b the sme fctor. (I.e. locl functon tht contns ll the prents z of x. Thus, f n ccompnng f(z, q lws exsts, we cn lws renormlze f(x, z. Snce the functons nd vrbles follow n cclc orderng n Besn network, chld vrbles cn be sequentll renormlzed bottom up wthout ffectng prevous normlztons. Fnll, the fctor contnng the vrble cn be renormlzed to preserve P ( x. The bove renormlzton strteg onl fls f, t n stge, the prent vrble set z s not contned n sngle locl functon, but s nsted splt between seprte locl functons. In ths cse, there would be no w to coordnte the compenston for ρ z (wthout ddng new locl functon over z. Thus, n the end, we re left wth n ntutve suffcent condton for when Besn network cn be renormlzed: An grph cn be normlzed wthout ffectng P ( xθ f the chld vrbles cn be elmnted wthout ddng n new edges. In these cses, we cn recover normlzed model wthout ffectng the optmlt of the soluton to (10, nd therefore we obtn globl mxmum of (8 wth respect to (3. 5 EXPERIMENTAL RESULTS p( jb x = δ jb (, x e δ(x w / e δ(x w Ths prml problem cn be solved n mn ws, ncludng tertve proportonl fttng. Thus, mn forms of prml-dul serch lgorthms re ble to effectvel solve the problem (10. We use strghtforwrd lterntng grdent pproch n our experments below. Of course, the solutons obtned to (10 m not be representble n Besn network becuse the prmeters w re sub-normlzed, not normlzed. The mn queston tht remns s when cn these subnormlzed solutons be converted nto properl normlzed Besn networks obeng the correct eqult constrnts (3? It turns out tht wde rnge of network topologes dmt smple procedure for renormlzng the locl functons so tht the become proper CPTs, wthout ffectng the condtonl probblt of gven x. In fct, ths observton hs been prevousl mde b (Wettg et l., 2002; Wettg et l., We present smpler vew here: It s es to chrcterze when n unnormlzed Besn network clssfer cn be renormlzed to preserve P ( x: Consder n unnormlzed locl functon f(x, z n Besn network structure, nd ssume we wnt to normlze t over x. Note tht ths functon cn lws be multpled b fctor ρ z for ech z, s long s there s nother locl functon f(z, q tht cn be To evlute the utlt of lernng mxmum mrgn Besn networks, we conducted some prelmnr experments on both rel nd snthetc dt sets. In the snthetc experments, we fxed Besn network structure nd prmeters, nd used t to generte trnng nd test dt. We expermented wth severl network topologes nd prmeterztons, nd compred mxmum mrgn Besn networks trned ccordng to (8 s.t. (3 to severl other pproches, ncludng: mxmum mrgn Mrkov networks (SVMs trned ccordng to (9 wth slcks, mxmum condtonl lkelhood (5, nd mxmum jont lkelhood (4. The results re for 20 repettons of the trnng smple, for the networks shown n Fgures 1 nd 2. Tbles 1 nd 2 show tht the technques behve smlrl, but show n dvntge for mxmrgbn over mxmrgmn. Ths mkes sense gven tht the dt ws generted from Besn network wth the sme structure consdered for trnng. The results show tht the onl technque whch gnores the Besn network normlzton constrnts, mxmrgmn, s slghtl behnd the other methods whch respect these constrnts, vndctng somewht the clm tht the normlzton structure of drected cusl model cn mpose n effectve mchne lernng bs, beond just provdng the fetures for generlzed lner model. We lso expermented wth rel dt from the UCI repostor. In these cses, we formulted Besn

5 Fgure 1: 8-node chn ugmented Nve Bes model. The clssfcton vrble s shded. network topolog tht ws ntended to cpture the cusl structure of the domn, but n ths cse hd no gurntee tht the presumed structure ws correct. These networks re much lrger nd cnnot be esl vsulzed here. We smpled 5 dsjont trnng sets out of ech dt set, tested on the remnder, nd report verge results. Tbles 3 7 show the results. Interestngl, these results generll show n dvntge for lernng technques tht respect the Besn network constrnts, nd dsdvntge for those tht gnore ths nformton. Surprsngl, mxmum jont lkelhood performed well n our experments. Unsurprsngl, mxmum condtonl lkelhood performed ver well. MxmrgBN performed best on one dt set. Tble 1: Accurc results for Fgure 1 lgorthm sze of trnng set mxl mxcl mxmrgbn mxmrgmn Fgure 2: 8-node twn-prent Nve Bes model. The clssfcton vrble s shded. Tble 2: Accurc results for Fgure 2 lgorthm sze of trnng set mxl mxcl mxmrgbn mxmrgmn MULTIVARIABLE EXTENSION Fnll, we extend the mxmum mrgn Besn network pproch to multvrble clssfcton. Ths ws the mn de of (Tskr et l., 2003; Altun et l., In ths settng, we observe trnng dt (x 1, 1,..., (x t, t. s before, but now the trgets re vectors of correlted clssfctons. Conceptull, ths extenson cuses no chnge n pproch, nd we cn seek to mxmze the crter (8 nd (9 s before. The onl new chllenge s copng wth the exponentl sum over. However, the dervton of our trnng lgorthm n Secton 4 s not sgnfcntl ffected b ths extenson. In fct, we fnd tht the dervtve L 1 / w jb now computes the mrgnl probblt over the locl vlues jb tht mtch the locl functon j on pttern b. We use stndrd probblstc nference technques (for exmple, forwrd-bckwrd to clculte these mrgnls, whch then llows us to clculte the grdents for the prml-dul optmzer. We mplemented ths pproch nd tested t on snthetc HMM model, where the clssfcton vrbles pl the role of the hdden stte sequence, nd the nput vrbles x pl the role of the observtons. We smpled (x, from 10 vrble HMM nd repeted the experment 20 tmes to obtn the fnl results. Tble 8 shows tht the mxmum mrgn pproch s competetve wth mxl nd mxcl, nd outperforms them t smple sze 100. Unfortuntel, t the tme of submsson we dd not hve multvrble verson of mxmum mrgn Mrkov networks vlble for comprson. Ths wll be dded. Nevertheless, the prelmnr results show credble performnce for mx mrgn Besn networks.

6 Tble 3: Accurc on UCI dt sets Austrln Brest Chess mxl mxcl mxmrgbn mxmrgmn Tble 8: Accurc on n 8 node HMM model lgorthm sze of trnng set mxl mxcl mxmrgbn CONCLUSION Tble 4: Accurc on UCI dt sets Corrl Crx Dbetes mxl mxcl mxmrgbn mxmrgmn Tble 5: Accurc on UCI dt sets Flre MofN Vote mxl mxcl mxmrgbn mxmrgmn Tble 6: Accurc on UCI dt sets Irs Vehcle Glss mxl mxcl mxmrgbn mxmrgnmn Tble 7: Accurc on UCI dt sets Lmphogrph Wveform-21 mxl mxcl mxmrgbn mxmrgnmn We hve nvestgted wht we feel s ver nturl queston; whether Besn network representton cn be combned wth dscrmntve trnng bsed on the mxmum mrgn crteron of SVMs. We hve found tht the outcome of ths nvestgton re mxed: Trnng Besn networks under the mxmum mrgn crteron s hrd computon problem hrder thn the stndrd qudrtc progrm of SVM trnng. However, resonble trnng lgorthms cn be devsed whch optmze the mrgn exctl n specl cses, but onl heurstcll n generl cses. On the other hnd, our prelmnr experments show tht there mght be n dvntge to respectng the cusl model constrnts emboded b Besn network, f ndeed these constrnts were present durng the dt generton. In ths sense, mx mrgn Bes nets offer new w to dd pror knowledge to SVMs. Unfortuntel, ths opportunt lso comes wth cost: mx mrgn Bes nets do not convenentl llow the kernel trck, whch loses one of the bggest dvntges of SVMs. In the end, t ppers tht mxmum mrgn Besn networks mght be vble lernng technque n multvrble clssfcton problems where there s strong pror cusl knowledge. However, ther utlt m be lmted b computtonl ntrctblt nd lck of kernel extenson. References Altun, Y., Tsochntrds, I., & Hofmnn, T. (2003. Hdden Mrkov support vector mchnes. Proceedngs Interntonl Conference on Mchne Lernng (ICML-03. Crmmer, K., & Snger, Y. (2001. On the lgorthmc nterpretton of multclss kernel-bsed vector mchnes. Journl of Mchne Lernng Reserch, 2. Fredmn, N., Geger, D., & Goldszmdt, M. (1997. Besn network clssfers. Mchne Lernng, 29,

7 Lffert, J., McCllum, A., & Perer, F. (2001. Condtonl rndom felds: Probblstc models for segmentng nd lbelng sequence dt. Proceedngs Interntonl Conference on Mchne Lernng (ICML-01. Ng, A., & Jordn, M. (2001. On dscrmntve vs. genertve clssfers. Advnces n Neurl Informton Processng Sstems 14 (NIPS-01. Schoelkopf, B., & Smol, A. (2002. Lernng wth kernels: Support vector mchnes, regulrzton, optmzton, nd beond. MIT Press. Tskr, B., Guestrn, C., & Koller, D. (2003. Mxmrgn Mrkov networks. Advnces n Neurl Informton Processng Sstems 16 (NIPS-03. Wettg, H., Grunwld, P., Roos, T., Mllmk, P., & Trr, H. (2002. On supervsed lernng of Besn network prmeters (Techncl Report HIIT Techncl Report Helsnk Insttute for Informton Technolog. Wettg, H., Grunwld, P., Roos, T., Mllmk, P., & Trr, H. (2003. When dscrmntve lernng of Besn network prmeters s es. Proceedngs Interntonl Jont Conference on Artfcl Intellgence (IJCAI-03.