Information Diffusion Kernels

Transcription

1 Information Diffusion Krnls John Laffrty Shool of Computr Sin Carngi Mllon nivrsity Pittsburgh P S laffrty@s.mu.du Guy Lbanon Shool of Computr Sin Carngi Mllon nivrsity Pittsburgh P S lbanon@s.mu.du bstrat nw family of krnls for statistial larning is introdud that xploits th gomtri strutur of statistial modls. Basd on th hat quation on th Rimannian manifold dfind by th Fishr information mtri information diffusion krnls gnraliz th Gaussian krnl of Eulidan spa and provid a natural way of ombining gnrativ statistial modling with non-paramtri disriminativ larning. s a spial as th krnls giv a nw approah to applying krnl-basd larning algorithms to disrt data. Bounds on ovring numbrs for th nw krnls ar provd using sptral thory in diffrntial gomtry and xprimntal rsults ar prsntd for txt lassifiation. 1 Introdution Th us of krnls is of inrasing importan in mahin larning. Whn krnlizd simpl larning algorithms an bom sophistiatd tools for takling nonlinar data analysis problms. Rsarh in this ara ontinus to progrss rapidly with most of th ativity fousd on th undrlying larning algorithms rathr than on th krnls thmslvs. Krnl mthods hav largly bn a tool for data rprsntd as points in Eulidan spa with th olltion of krnls mployd limitd to a fw simpl familis suh as polynomial or Gaussian RBF krnls. Howvr rnt work by Kondor and Laffrty [7] motivatd by th nd for krnl mthods that an b applid to disrt data suh as graphs has proposd th us of diffusion krnls basd on th tools of sptral graph thory. On limitation of this approah is th diffiulty of analyzing th assoiatd larning algorithms in th disrt stting. For xampl thr is no obvious way to bound ovring numbrs and gnralization rror for this lass of diffusion krnls sin th natural funtion spas ar ovr disrt sts. In this papr w propos a rlatd onstrution of krnls basd on th hat quation. Th ky ida in our approah is to bgin with a statistial modl of th data bing analyzd and to onsidr th hat quation on th Rimannian manifold dfind by th Fishr information mtri of th modl. Th rsult is a family of krnls that naturally gnralizs th familiar Gaussian krnl for Eulidan spa and that inluds nw krnls for disrt data by bginning with statistial familis suh as th multinomial. Sin th krnls ar intimatly basd on th gomtry of th Fishr information mtri and th hat or diffusion quation on th assoiatd Rimannian manifold w rfr to thm as information diffusion krnls.

2 : : _ ^ ^ nlik th diffusion krnls of [7] th krnls w invstigat hr ar ovr ontinuous paramtr spas vn in th as whr th undrlying data is disrt. s a onsqun som of th mahinry that has bn dvlopd for analyzing th gnralization prforman of krnl mahins an b applid in our stting. In partiular th sptral approah of Guo t al. [3] is appliabl to information diffusion krnls and in applying this approah it is possibl to draw on th onsidrabl body of rsarh in diffrntial gomtry that studis th ignvalus of th gomtri Laplaian. In th following stion w rviw th rlvant onpts that ar rquird from information gomtry and lassial diffrntial gomtry dfin th family of information diffusion krnls and prsnt two onrt xampls whr th undrlying statistial modls ar th multinomial and sphrial normal familis. Stion 3 drivs bounds on th ovring numbrs for support vtor mahins using th nw krnls adopting th approah of [3]. Stion 4 dsribs xprimnts on txt lassifiation and Stion 5 disusss th rsults of th papr. 2 Information Gomtry and Diffusion Krnls Lt b a -dimnsional statistial modl on a st. For ah "!#$ % assum th mapping is &(' at ah point in th intrior of. Lt )+* - /.10 and Th Fishr information matrix 9 : *< of at is givn by or quivalntly as *< =B. 9 ) * 2. ) 2. >CEDGF ) * ) 3H5I78 %/ % (1) *J KLNM DGF )+*1O % )PO (Q * (2) In oordinats : *< = dfins a Rimannian mtri on giving th strutur of a -dimnsional Rimannian manifold. On of th motivating proprtis of th Fishr information mtri is that unlik th Eulidan distan it is invariant undr rparamtrization. For dtaild tratmnts of information gomtry w rfr to [1 6]. For many statistial modls thr is a natural way to assoiat to ah data point a pa- in th statistial modl. For xampl in th as of txt undr th multinomial modl a doumnt is naturally assoiatd with th rlativ frqunis of th word ounts. This amounts to th mapping whih snds a doumnt to its maximum S liklihood modl R. Givn suh a mapping w propos to apply a krnl on paramtr spa T@ SV XW T@ = 1V 1. ramtr vtor undr a suitabl prior. In th as of txt this is on way of smoothing th maximum liklihood modl using for xampl a Dirihlt prior. Givn a krnl on paramtr spa w thn avrag ovr th postriors to obtain a krnl on data: Mor gnrally w may assoiat a data point with a postrior distribution =Z T@ V X DG[\DG[ T@ =]1 V / =Z / K V V V Q (3) It rmains to dfin th krnl on paramtr spa. Thr is a fundamntal hoi: th krnl assoiatd with hat diffusion on th paramtr manifold undr th Fishr information mtri. For a manifold ^ with mtri : *J th Laplaian _a`sbl d# bl is givn in loal oordinats by f dt:@g *J ) * O dt:x: *J ) (4)

3 : * [! * 3 d *J whr 9 : >8 9 : *J > * 0. Whn ^ gnralizing th lassial oprator div is ompat th Laplaian has disrt ignvalus * with orrsponding ignfuntions satisfying _ *. Whn th manifold has a boundary appropriat boundary onditions must b imposd in ordr that _ is slf-adjoint. Dirihlt boundary * onditions st /0 [ and Numann boundary onditions rquir whr is th outr normal dirtion. Th following thorm summarizs th basi proprtis for th krnl of th hat quation _ on ^. "!] "!] #! "!]+* )!G (3) _ a.- T (4) "!] 0/ [ T T 1 534!] and (5) "!] *'6 ' 87 :9 0 * * #!]. W rfr to [9] for a proof. Proprtis 2 and 3 imply that 4!] solvs th hat quation 7! in starting from. Intgrating proprty 3 against a funtion #!] shows that #< =/ [ "!] )!] / [ / [. Thrfor 4!G #!]! / [ 7 #< -?> 7 #< :@B 7 sin #< is a positiv oprator thus 4!G is positiv dfinit. Togthr ths proprtis show that dfins a Mrr! krnl. Not that whn using suh a krnl for lassifiation th disriminant funtion *DC *! * T@! * * X an b intrprtd as th solution to th hat quation with initial tmpratur on labld data point C *! * *! and Thorm 1. Lt ^ b a godsially omplt Rimannian manifold. Thn th hat krnl T@ xists and satisfis (1) T@ 3%$'& (2) )( T@ on unlabld points. Th following two basi xampls illustrat th gomtry of th Fishr information mtri and its assoiatd diffusion krnl: th multinomial orrsponds to a Rimannian manifold of onstant positiv urvatur and th sphrial normal family to a spa of onstant ngativ urvatur. 2.1 Th Multinomial Th multinomial is an important xampl of how information diffusion krnls an b 1 applid naturally to disrt data. For th multinomial family is an lmnt of th *'6 FE -simplx * *!#HG f * I3 *. Th transformation maps th -simplx to th -sphr of radius 2. Th rprsntation of th Fishr information mtri givn in quation (2) suggsts th gomtry undrlying th multinomial. In partiular th information mtri is givn by *J K FE J 6 J ) * 3H5I7 J ) 3H5I7X J K> ) * 3 ) so that th Fishr information orrsponds to th innr produt of tangnt vtors to th sphr and information gomtry for th multinomial is th gomtry of th positiv orthant of th sphr. Th godsi distan btwn two points ] V is givn by K] V XLGNMDOFP.P 5RQTS FE g*%6 O * *5 V Q (5) This mtri plas gratr mphasis on points nar th boundary whih is xptd to b important for txt problms whih hav spars statistis. In gnral for th hat krnl on a Rimannian manifold thr is an asymptoti xpansion in trms of th paramtris s for xampl [9]. This xpands th krnl as 4!GL KMWV:X1ZT[]\_^a` "!] MbX 0d g*'6 f 4!] X *gih #X sing th first ordr approximation and th xpliit distan for th godsi distan givs (6)

4 G V G V S & g Figur 1: Exampl dision boundaris using support vtor mahins with information diffusion krnls for trinomial gomtry on th 2-simplx (top right) and sphrial normal gomtry LG (bottom right) ompard with th standard Gaussian krnl (lft). a simpl formula for th approximat information diffusion krnl for th multinomial as K] V KMbV:X1 []\_^ S X G MbO P P 5WQ FE g*%6 O * *) V In Figur 1 this krnl is ompard with th standard Eulidan spa Gaussian krnl for th as of th trinomial modl. 2.2 Sphrial Normal /% 81 Now onsidr th statistial family givn by N whr is th man and is th sal of th varian. alulation shows that : *J K * *J. Thus th Fishr information mtri givs E th strutur of th uppr half plan in hyprboli spa. Th hat krnl on hyprboli spa by and for V g G G V X SV f MbV:X ` th krnl is givn by f G f MbV:X ` Q"$ has a losd form [2]. For Q4$ ) ) 8 ) ) D ' []\_^ ` X G [.\_^! #" E %$ 1 & (' f P 5RQ% P 5WQ)* (7) it is givn MWX_ (8) whr is th godsi distan btwn th two points in. For th krnl is idntial to th Gaussian krnl on. If only th man + is unspifid thn th assoiatd krnl is th standard Gaussian RBF krnl. In Figur 1 th krnl for hyprboli spa is ompard with th Eulidan (9)

5 ' " ^ % g! ' G spa Gaussian krnl for th as of a 1-dimnsional normal modl with unknown man and varian orrsponding to. Not that th urvd dision boundary for th diffusion krnl maks intuitiv sns sin as th varian drass th man is known with inrasing rtainty. 3 Sptral Bounds on Covring Numbrs In this stion w prov bounds on th ntropy and ovring numbrs for support vtor mahins that us information diffusion krnls ths bounds in turn yild bounds on th xptd risk of th larning algorithms. W adopt th approah of Guo t al. [3] and mak us of bounds on th sptrum of th Laplaian on a Rimannian manifold rathr than on VC dimnsion thniqus. Our alulations giv an indiation of how th undrlying gomtry influns th ntropy numbrs whih ar invrs to th ovring numbrs. X W bgin by ralling th main rsult of [3] modifying thir notation slightly to onform with ours. Lt ^ b a ompat subst of -dimnsional Eulidan spa and suppos that T `S^ S#$ is a Mrr krnl. Dnot by I th ignvalus of T i.. of th mapping!#/ [ 1 64!G #!] < and lt dnot th df LQ orrsponding ignfuntions. W assum that & ^. Givn * points ^ * th SVM hypothsis lass for with wight vtor boundd by is dfind as th olltion of funtions X Q Q Q!#$"> d Q Q Q > d d1 (10) whr > H is th mapping from ^ to fatur spa dfind by th Mrr krnl and dnot th orrsponding Hilbrt spa innr produt and norm. It is of intrst to obtain uniform bounds on th ovring numbrs 1 dfind as th siz of th smallst -ovr of in th mtri indud by th norm &M ' \ *%6 * Thorm 2. E Q ^)( 0'*+ [- 9! 9#".. Th following is th main rsult of Guo t al. [3]. lt and dfin $ & Givn an intgr 1. dnot th smallst intgr for whih 9 9 "'& " & g *%6S ' & * Q Thn To apply this rsult w will obtain bounds on th indis / using sptral thory in Rimannian gomtry. Th following bounds on th ignvalus of th Laplaian ar du to Li and au [8]. Thorm 3. Lt ^ b a ompat Rimannian manifold of dimnsion with non-ngativ Rii urvatur and assum that th boundary of ^ is onvx. Lt dnot th ignvalus of th Laplaian with Dirihlt boundary onditions. Thn 0 0 ` (11) ` 1 whr 1 is th volum of ^ and 0 and 0 ar onstants dpnding only on th dimnsion. Not that th manifold of th multinomial modl satisfis th onditions of this thorm. sing ths rsults w an stablish th following bounds on ovring numbrs for information diffusion krnls. W assum Dirihlt boundary onditions a similar rsult an b provn for Numann X boundary onditions. W inlud th onstant 1 vol ^ and diffusion offiint in ordr to indiat how th bounds dpnd on th gomtry. 1

6 ^ 0 / - Thorm 4. Lt ^ b a ompat Rimannian manifold with volum 1 satisfying th onditions of Thorm 3. Thn th ovring numbrs for th Dirihlt hat krnl T( on satisfy Proof. 3H5I7 1L h `T` 1 X 3H5I7 FE ` T (12) By th lowr bound in Thorm 3 th Dirihlt ignvalus of th hat krnl T@ "!] 7 whih ar givn by " 3657 satisfy X 0. Thus 3H5I7 F X 0 g G ` ` 1 3H5I7 X 0 g G ` g G (13) g*'6 *'6 whr th sond inquality oms from E. Now using th uppr bound of Thorm 3 th inquality / X 0 ` g G 1 or quivalntly X 0 1 ` 3H5I7 E g GI 0 Th abov inquality will hold in as S X 0 G 1 0 FE 3657 FE will hold if X 0 g G FE S g G ` 1 G3657 g G 1 g G X (14) (15) sin w may assum that 0 0 thus 0 ` 3657 onstant 0. Plugging this bound on into th xprssion for *'6 and using ' & 7 * h 7 & 3657 ' w hav aftr som algbra that ` 3H5I7 ' FE (16) for a nw ' in Thorm 2. Invrting th abov quation in 3657 givs quation (12) ' and for fixd 3H5I7 thy sal h X ' in th diffusion W not that Thorm 4 of [3] an b usd to show that this bound dos not in fat h 3657 dpnd on and. Thus for fixd X th ovring numbrs sal as tim X. 4 Exprimnts W ompard th information diffusion krnl to linar and Gaussian krnls in th ontxt of txt lassifiation using th WbKB datast. Th WbKB olltion ontains som 4000 univrsity wb pags that blong to fiv atgoris: ours faulty studnt projt and staff. bag of words rprsntation was usd for all thr krnls using only th word frqunis. For simpliity all hyprtxt information was ignord. Th information diffusion krnl is basd on th multinomial modl whih is th orrt modl undr th

7 #! R R linar rbf diffusion 0.3 linar rbf diffusion Tst st rror rat Tst st rror rat Numbr of training xampls Numbr of training xampls Figur 2: Exprimntal rsults on th WbKB orpus using SVMs for linar (dot-dashd) and Gaussian (dottd) krnls ompard with th information diffusion krnl for th multinomial (solid). Rsults for two lassifiation tasks ar shown faulty vs. ours (lft) and faulty vs. studnt (right). Th urvs shown ar th rror rats avragd ovr 20-fold ross validation. (inorrt) assumption that th word ourrns ar indpndnt. Th maximum liklihood mapping was usd to map a doumnt to a multinomial modl simply normalizing th ounts to sum to on. Figur 2 shows tst st rror rats obtaind using support vtor mahins for linar Gaussian and information diffusion krnls for two binary lassifiation tasks: faulty vs. ours and faulty vs. studnt. Th urvs shown ar th man rror rats ovr 20-fold ross validation and th rror bars rprsnt twi th standard dviation. For th Gaussian and information diffusion krnls w tstd valus of th krnls fr paramtr ( f X or ) in th st Q Q GG QG G] +. Th plots in Figur 2 us th bst paramtr valu in th abov rang. Our rsults ar onsistnt with prvious xprimnts on this datast [5] whih hav obsrvd that th linar and Gaussian krnls rsult in vry similar prforman. Howvr th information diffusion krnl signifiantly outprforms both of thm almost always obtaining lowr rror rat than th avrag rror rat of th othr krnls. For th faulty vs. ours task th rror rat is halvd. This rsult is striking baus th krnls us idntial rprsntations of th doumnts vtors of word ounts (in ontrast to for xampl string krnls). W attribut this improvmnt to th fat that th information mtri plas mor mphasis on points nar th boundary of th simplx. 5 Disussion Krnl-basd mthods gnrally ar modl fr and do not mak distributional assumptions about th data that th larning algorithm is applid to. t statistial modls offr many advantags and thus it is attrativ to xplor mthods that ombin data modls and purly disriminativ mthods for lassifiation and rgrssion. Our approah brings a nw prsptiv to ombining paramtri statistial modling with non-paramtri disriminativ larning. In this aspt it is rlatd to th mthods proposd by Jaakkola and Hausslr [4]. Howvr th krnls w invstigat hr diffr signifiantly from th Fishr krnl proposd in [4]. In partiular th lattr is basd on th Fishr sor. at a singl point R givn by a ovarian SV d * T *. 9 * > - V* Ä. 9 * > in paramtr spa and in th as of an xponntial family modl it is -. In ontrast infor-

8 R mation diffusion krnls ar basd on th full gomtry of th statistial family and yt ar also invariant undr rparamtrization of th family. Bounds on th ovring numbrs for information diffusion krnls wr drivd for th as of positiv urvatur whih apply to th spial as of th multinomial. W not that th rsulting bounds ar ssntially th sam as thos that would b obtaind for th Gaussian krnl on th flat -dimnsional torus whih is th standard way of ompatifying Eulidan spa to gt a Laplaian having only disrt sptrum th rsults of [3] ar formulatd for th as orrsponding to th irl. Similar bounds for gnral manifolds with urvatur boundd blow by a ngativ onstant should also b attainabl. Whil information diffusion krnls ar vry gnral thy may b diffiult to omput in partiular ass xpliit formulas suh as quations (8 9) for hyprboli spa ar rar. To approximat an information diffusion krnl it may b attrativ to us th paramtris and godsi distan K1 V btwn points as w hav don for th multinomial. In ass whr th distan itslf is diffiult to omput xatly a ompromis may b to approximat th distan K] btwn narby points in trms of th Kullbak-Liblr divrgn V 1 % V. using th rlation Th primary dgr of frdom in th us of information diffusion krnls lis in th. For th multinomial!# IMDO 7 &M w hav usd th maximum liklihood mapping \. % whih is simpl and wll motivatd. s indiatd in Stion 2 thr ar othr possibilitis. This rmains an intrsting ara to xplor partiularly for latnt variabl modls. spifiation of th mapping of data to modl paramtrs!# knowldgmnts This work was supportd in part by NSF grant CCR Rfrns [1] S. mari and H. Nagaoka. Mthods of Information Gomtry volum 191 of Translations of Mathmatial Monographs. mrian Mathmatial Soity [2]. Grigor yan and M. Noguhi. Th hat krnl on hyprboli spa. Bulltin of th London Mathmatial Soity 30: [3]. Guo P. L. Bartltt J. Shaw-Taylor and R. C. Williamson. Covring numbrs for support vtor mahins. IEEE Trans. Information Thory 48(1) January [4] T. S. Jaakkola and D. Hausslr. Exploiting gnrativ modls in disriminativ lassifirs. In dvans in Nural Information Prossing Systms volum [5] T. Joahims N. Cristianini and J. Shaw-Taylor. Composit krnls for hyprtxt atgorisation. In Prodings of th Intrnational Confrn on Mahin Larning (ICML) [6] R. E. Kass and P. W. Vos. Gomtrial Foundations of symptoti Infrn. Wily Sris in Probability and Statistis. John Wily & Sons [7] R. I. Kondor and J. Laffrty. Diffusion krnls on graphs and othr disrt input spas. In Prodings of th Intrnational Confrn on Mahin Larning (ICML) [8] P. Li and S.-T. au. Estimats of ignvalus of a ompat Rimannian manifold. In Gomtry of th Lapla Oprator volum 36 of Prodings of Symposia in Pur Mathmatis pags [9] R. Shon and S.-T. au. Lturs on Diffrntial Gomtry volum 1 of Confrn Prodings and Ltur Nots in Gomtry and Topology. Intrnational Prss 1994.