Mathematical Issues. Written by Okito Yamashita, Last revised 2009/07/27

Transcription

1 Mahemaical Issues Wrie by Okio Yamashia, Las revise 009/07/7 I SLR oolbox, here are seve biary classificaio algorihms implemee. I his ocume, he probabilisic moels of he implemee classifiers are escribe a he he erivaio of algorihms will be briefly summarize. 0. oaios - aa {( x, y ),,( x, y )} x R, y {0,} i i X = [ x,, x ], y = [ y,, y ] - Liear iscrimia fucio = f( xw ; ) = w x + w = wx where w = [ w0, w,, w ], x [, x,, x] - Sigmoi fucio (Logisic fucio) 0 ( x ) σ ( x) = / + exp( ) - Liear logisic regressio moel P( X, ) P( y, ) ( ) y w y y = x w = σ σ () where σ = σ ( wx ). - ormal isribuio ( ;, ) exp ( ) ( ) x µ S = π S S x µ x µ E( x) = µ, V( x) = S - Gamma isribuio 0 0 γ 0 γ 0 γ 0 γ 0 α α Γ( γ0) α0 α0 Γ ( αα ;, γ) = exp 0 E( α) = α0, V( α) = α γ 0

2 . Probabilisic Moel All he probabilisic moels irouce here ca be escribe i he Bayesia moel, which has he likelihoo fucio a a prior isribuio for weigh (or bouary) parameers. The likelihoo fucios of all he moels are he ieical fucio kow as he logisic regressio moel (Eq.()) while he prior isribuios are iffere forms.. Sparse Logisic Regressio (SLR) The likelihoo fucio is give by Eq.(). The prior isribuio of SLR has he followig hierarchical form; For =,, P0 ( w α) = (0, α ) P0 ( α) = α () This hierarchical isribuios are kow as he auomaic relevace eermiaio (AR) priors i he sparse Bayesia learig lieraure. The parameer α is calle he relevace parameer ha represes relevace of he correspoig weigh parameer. The larger his value is, less releva he correspoig weigh parameer is. If he relevace parameer is margialize beforeha, we kow ha he hierarchical prior isribuios are equivale o he followig prior isribuio of weigh parameers, P( w ) = / w =,,. 0 The hierarchical form makes i easier o calculae he poserior isribuio i aalyical way. Thus he equaio () is use for represeig he AR prior.. Regularize Logisic Regressio (RLR) The likelihoo fucio is give by Eq.(). The prior isribuio of RLR is he followig mulivariae Gaussia isribuio; P w α I. (3) ( ) (0, ) 0 = I is he ieiy marix of size..3 Liear Relevace Vecor Machie (RVM)

3 The likelihoo fucio is give by Eq.() excep he liear iscrimia fucio beig represee by he liear kerel as follows, = g( xw ; ) = w xx+ w = k ( x) w 0 where ( ) = (,,,) kx xx xx a w = ( w,, w, w0). Thus he likelihoo for each sigle aa is represee by σ = σ ( kx ( ) w). The prior isribuio of RVM is he AR prior (Eq()). I shoul be oe ha he sparseess of SLR a RVM resuls i he iffere ierpreaio. The parameers i SLR are associae wih feaures, while he parameers i SVM are associae wih samples (oe ha he umber of weigh parameers i RVM is o ). If he parameers i SLR are esimae i a sparse way, i ca be ierpree as feaure selecio process bu ha of RVM ca be ierpree as sample selecio process (similar o suppor vecors i SVM). Aoher hig o be oice is ha he exesio o he oliear iscrimia fucio is easily realize i RVM because RVM uses he kerel represeaio. Chagig he liear kerel o ay oher oliear kerel such as Gaussia kerel, polyomial kerel ca moel a o-liear bouary..4 L-Sparse Logisic Regressio (L-SLR) The likelihoo fucio is give by Eq.(). The prior isribuio of L-SLR is he followig Laplace isribuio; P0 ( w) = λ exp( λ w ) =,,. (4) This Laplace prior ca be expresse i he hierarchical form as follows; P0 ( w α) = (0, α) λ λ P0 ( α) = exp α =,, (5) oe ha α is variace of Gaussia isribuio i Eq.(5) whereas i is precisio (iverse variace) of Gaussia isribuio i Eq.().

4 . erivaio of Algorihms Sice all he probabilisic moels escribe above are Bayesia moels, he ask o esimae weigh parameers is o calculae he poserior probabiliy isribuio of weigh parameers give by P( y w) P0( w α) P0( α) α P( w y) = P( y w) P( w α) P( α) αw. 0 0 Here a hereafer he epeecy for X is igore for oaioal simpliciy. Uforuaely he iegrals of umeraor a eomiaor are o aalyically racable. Therefore some approximaio meho shoul be applie. I he algorihm erivaio, we apply he variaioal Bayesia meho (VB) ha assumes he coiioal iepeece coiio o poserior isribuios a he solve he poserior calculaio by maximizig a specific crieria (calle free eergy). A firs VB efies he free eergy usig he es fucio Q(); P( ywα,, ) FE( Q( w, α)) = Q( w, α)log αw. (6) Q( w, α) The oable issue i his equaio is ha FE is maximize whe a oly whe he es fucio Q is equal o he joi poserior isribuio P( w, α y ). I aiio he maximize value is equivale o he eviece P( y ). Therefore maximizig FE wih respec o es fucio correspos o fiig he joi poserior isribuio a compuig he eviece. However his fucioal maximizaio is as ifficul as he origial problem, hus i ca o be solve irecly. VB solves his problem by resricig he es fucio o havig some fucioal form. I our applicaio, he es fucio assumes o saisfy he coiioal iepeece coiio as Q( w, α) = Q( w) Q( α ). (7) Uer his coiio, FE is maximize by aleraely maximizig FE wih respec o Q( w ) a Q( α ). These seps are equivale o compuig he followig W-sep a A-sep (for more mahemaical eails, see Bishop 006). W-sep : log Q( w) =< log P( y, w, α ) > Q ( α) A-sep : log Q( α) =< log P( ywα,, ) > Q ( w) where < f ( x ) > Q ( x) eoes he expecaio of f ( x ) wih respec o a probabilisic

5 measure Q( x ). Cocree algorihms o compue he poserior isribuios as well as he poserior mea (he esimae of he weighs) ca be erive by subsiuig he probabilisic moels escribe i secio. Bu uforuaely eve wih his approximaio, W-sep is he aalyically iracable for he logisic regressio moel sice he prior isribuio a he likelihoo fucio are o cojugae. A furher approximaio o Q( w ) is require. There are wo approximaio mehos for his purpose; Laplace approximaio a variaioal approximaio. Boh of he mehos use he Gaussia isribuio as he approximae isribuio. This is why here are wo algorihms i he oolbox eve for oe probabilisic moel SLR (also RLR)... SLR... SLR wih Laplace approximaio (SLR-LAP) This algorihm uses he Laplace approximaio ha approximaes he poserior isribuio wih Gaussia isribuio arou he MAP esimae. W-sep: log Q( w) = { y log σ + ( y) log( σ) } w Aw + cos (8) = where A= iag( α,, α ). Le s eoe he righ-ha sie of Eq (8) by E( w ). E( w ) is o he quaraic form of w, hus Q( w ) is o he Gaussia isribuio a aalyically iracable. Bu if we oice ha E( w ) is he exacly same form as he log-likelihoo fucio of logisic regressio wih he regularizaio erm, his fucio is well approximae by he quaraic fucio a he maximizer w (Laplace approximaio). This maximizaio is oe by he ewo-rapso meho usig he followig graie a Hessia E = Xδ Aw, w E = XBX ' A H( w) ww where δ, X, B are give by δ = [ y σ,, y σ] : X = [ x,, x ] :. B = iag( σ ( σ ),, σ ( σ )) : The we obai he quaraic approximaio of log Q( w ) (or equivalely E( w ))

6 arou he maximizer w, log Q( w) E( w) ( w w) H( w)( w w ). (9) Thus Q( w) ( w, S) where S H( w ). A-sep: log Q( α ) = ( α ( w + s ) + logα ) + cos Thus = Q( α ) = Q( α ) = Γ( α ; α, γ ) where α =, = w + s γ (0) = = w, s are he h eleme a he h iagoal eleme of w a S, respecively. This upaig rule is very slow o coverge. Isea we use he followig fas upaig rule moivae by Mackay s effecive egree of freeom. Fas upaig rule: Q( α) = Γ( α ; α, γ ) where α = γ =. () = αs, w... SLR wih variaioal approximaio (SLR-VAR) This algorihm firs approximae he logisic fucio wih a Gaussia isribuio wih oe auxiliary variable (variaioal approximaio). Accorig o Jaakkola a Jora 000, he logisic fucio ca be lower-boue by x ξ σ( x) σ( ξ)exp λ( ξ)( x ξ ) where λξ ( ) = σξ ( ) > 0 ( ξ> 0). ξ Thus he likelihoo fucio is boue by z ξ P( y w) σξ ( )exp { λξ ( )( z ξ ) z( y) } h( y, w, ξ) wih z = xw. Usig his formula, FE is lower-boue as follows, h( ywξ,, ) P0( w α) P0( α) FE( Q( w, α)) (, )log Q w α αw FE( Q( w, α), ξ). Q( w, α) Thus maximizig FE ( Q( w, α), ξ ) wih respec o Q( w), Q( α), ξ aleraely gives us he poserior isribuios a opimal variaioal parameers ξ. Sice he fucio h( ywξ,, ) is he quaraic fucio of w for fixe ξ, Q( w ) becomes he Gaussia isribuio, which is easy o calculae aalyically. See Bishop a Tippig 000 for more eails

7 W-sep: log Q( w) = w A + ( ) x x w+ ( y 0.5) x w+ cos λξ = = ( ) ( ) = w w S w w + cos Thus Q( w) ( w, S) where w = S( y 0.5) x : = X = [ x,, x ] : S = A+ λξ ( ) xx = A+ XΛ X : = Λ= iag( λξ ( ),, λξ ( )) : () A-sep: A-sep is ieical o ha of SLR-LAP (see Eq.()). Fas upaig rule : Q( α ) = Q( α ) = Γ( α ; α, γ ) where α = γ = (3) = = αs, w -sep: Takig parial erivaive of FE ( Q( w, α), ξ ) a seig i o zero, we have ξ = x ( ww + S) x =,, (4).. RLR... RLR wih Laplace approximaio (RLR-LAP) W-sep: If = (,, ) A iag α α i SLR-LAP algorihm is replace wih A = α I is ieical o SLR-LAP algorihm., his sep A-sep: log Q( α) = < ww > α + logα + cos

8 Thus Q( α) =Γ( α ; α, γ ) where α =, γ = (5) w + s ( ) = α s = Fas upaig rule : Q( α) =Γ( α ; α, γ ) where α =, γ =.(6) w... RLR wih variaioal approximaio (RLR-VAR) W-sep: If = (,, ) A iag α α i SLR-VAR algorihm is replace wih A = α I is ieical o SLR-VAR algorihm. =, his sep A-sep: A-sep is equivale o ha of RLR-LAP. α s = Fas upaig rule : Q( α) =Γ( α ; α, γ ) where α =, γ =. w -sep: This sep is ieical o ha of SLR-VAR sice his sep oes o epe o he prior isribuio. ξ = x ( ww + S) x =,, =.3 RVM The algorihm implemee for RVM is ieical o SLR-VAR excep he bouary beig moele by he liear kerels. Thus if z = xw i SLR-VAR algorihm is replace wih z = k ( x ) w, he erivaio is ieical. See Tippig 00 for he origial RVM algorihm. W-sep:

9 log Q( w) = w A + ( ) k( x ) k ( x ) w+ ( y 0.5) k ( x ) w+ cos λξ = = ( ) ( ) = w w S w w + cos Thus Q( w) ( w, S) where S = A+ λξ ( ) kx ( ) k( x) = A+ XΛ X : ( + ) ( + ) = w = S( y 0.5) k( x) : ( + ) = (7) X = [ kx ( ),, kx ( )] : ( + ) Λ= iag( λξ ( ),, λξ ( )) : A-sep: A-sep is ieical o ha of SLR-LAP (see Eq.()). Fas upaig rule: = = αs, w Q( α ) = Q( α ) = Γ( α ; α, γ ) where α = γ = (8) -sep: Takig parial erivaive of FE ( Q( w, α), ξ ) a seig i equal o zero, we have ξ = k ( x )( ww + S) k( x ) =,,. (9).4 L-SLR The algorihm of L-SLR-LAP is obaie by moifyig he erivaio i Krishapuram e.al 004. The algorihm of L-SLR-c is base o compoe-wise upae proceure wih he surrogae fucio i Krishapuram e.al L-SLR wih Laplace approximaio (L-SLR-LAP) We use he hierarchical form of he Laplace prior (Eq.5). The erivaio is very similar o ha of SLR-LAP. W-sep:

10 log Q( w) = { y log σ + ( y) log( σ) } wvw + cos (0) = where V = iag ( < α >,, < α > ). Q( w ) is obaie i he same way (ewo-rapso meho) as W-sep of SLR-LAP wih A replace wih V. A-sep: + log ( ) = w s Q α λα logα + cos = α Here w, s eoe he h eleme a he h iagoal eleme of he weigh poserior isribuio. = Q( α) = Q( α ). () w λ + s Q( α) = Cα exp α α Uforuaely his isribuio is o he Gamma isribuio ulike SLR moel. Bu oly ecessary quaiy i W-sep is he expecaio of α a his value ca be aalyically compue as follows, < >= α α Q( α) α = = = α α α 3 w + s λ exp α w + λ s exp α α α α π λ w + s exp w + s π λ w + exp s λ λ w + s () The formula erive from EM algorihm is very similar (see Krishapuram e.al

11 004), < α >= λ w. (3) Oly he ifferece is wheher he poserior variace of weighs is iclue i he eomiaor. My lile experiece showe ha Eq.(3) oes work well bu Eq.3 oes o. Thus Eq.(3) is applie i he curre implemeaio. Reasos for his issue have o bee mae clear ye..4.. L-SLR wih compoe-wise upaes (L-SLR-c) This algorihm uses he prior isribuio Eq.(4) raher ha he hierarchical prior isribuios Eq.(5). Thus he relevace parameers o o appear i he algorihm erivaio. The weigh upaig rule is obaie by irecly iffereiaig he surrogae fucio. The MAP esimae of weigh parameers are obaie by { P } w = arg max log ( y w) λ w. l Takig he lower bou of he log P( y w ), we have he followig surrogae fucio, Q( w wˆ ) = w ( g( wˆ ) Bwˆ ) + w Bw λ w () () () The fucio g () is he graie of log P( y w ). The marix B ca be ay posiive-efiie marix ha bous Hessia of log P( y w) everywhere. Here we chose 0.5 B = X ' X 0.5 where X = [ x,, x ] :. l The by irecly iffereiaig ˆ () Q( w w ) wih respec o w we have he followig upaig rule,

12 () ( ) ( ) ( ˆ ) λ + + gk w w = sof w ; Bkk Bkk where sof( a; δ ) = sig( a)max{0, a δ } (4) For more mahemaical eails, please rea he origial paper.

13 3. Algorihm Summary The algorihms of SLR-LAP a SLR-VAR are escribe here. The oher algorihms (RLR-LAP, RLR-VAR, RVM, L-SLR-LAP) are similar herefore omie. - SLR-LAP. Iiializaio Se α = =,,. W-sep Q( w) ( w, S) where S H( w ) w is he maximizer of E( w ) ha ca be obaie by he ewo-rapso meho. ( ) = E w { ylog σ + ( y)log( σ) } w Aw + cos = where E = Xδ Aw w, E = XBX ' A H( w) w w A= iag( α,, α) : δ = [ y σ,, y σ] :. X = [ x,, x ] : B = iag( σ ( σ ),, σ ( σ )) : 3. A-sep: Q( α ) = Q( α ) = Γ( α ; α, γ ) = = αs, w α = γ = ( see origial rule : α =, = w + s γ ) w, s are he h eleme a he h iagoal eleme of w a S, respecively. 4. Covergece: Coiue W-sep a A-sep aleraely uil he amou of weigh parameers is very small or he umber of ieraios excees he preefie value.

14 - SLR-VAR. Iiializaio Se α = =,, a ξ = =,,. W-sep Q( w) ( w, S) w = S( y 0.5) x : S = A+ λξ ( ) xx = A+ XΛ X : = = ( + Λ ) y < = A X X X A X( Λ + X A X) Λ y > y = [ y 0.5,, y 0.5] : X = [ x,, x ] : Λ= iag( λξ ( ),, λξ ( )) : λξ ( ) = σξ ( ) > 0 ( ξ> 0) ξ 3. A-sep: Q( α ) = Q( α ) = Γ( α ; α, γ ) = = αs, w α = γ = ( see origial rule : α =, = w + s γ ) w, s are he h eleme a he h iagoal eleme of w a S, respecively. 4. -sep: ξ = x ( ww + S) x =,, 5. Covergece: Coiue W-sep, A-sep a -Sep aleraely uil he amou of weigh parameers is very small or he umber of ieraios excees he preefie value.

15 Refereces Jaakkola TS, Jora MI (000), Bayesia parameer esimaio via variaioal mehos, Saisics a Compuig, 0, pp.5 37 Bishop C, Tippig ME (000),Variaioal relevace vecor machies, Proceeigs of he 6h Coferece i Uceraiy i Arificial Ielligece, pp Bishop C (006), Paer recogiio a machie learig, Spriger, ew York Tippig ME(00), Sparse Bayesia Learig a he Relevace Vecor Machie, J Machie Learig Research,, pp.-44 Krishapuram B, Haremik AJ, Cari L a Figueireo MAT (004), A Bayesia Approach o Joi Feaure Selecio a Classifier esig, IEEE Tras. Paer Aalysis a Machie Ielligece, 6, pp.05- Krishapuram B, Cari L, Figueireo MAT, a Haremik AJ (005), Sparse Muliomial Logisic Regressio Fas Algorihms a Geeralizaio Bous, IEEE Tras. Paer Aalysis a Machie Ielligece, 7, pp MacKay (99), Bayesia ierpolaio, eural Compuaio, 4, pp Appeix : Trasformaio of equaios : eails This appeix preses eails of rasformaio of equaios. This may be useful o wrie your ow coes. I. Sparse Logisic Regressio (SLR) P( w) = P( w α) P( α) α α = ( ) exp π α w α α α = ( ) exp π α w α w w α π α α = ( ) Γ(/) exp w Γ(/) w = ( π) π = w

16 I.4 L-Sparse Logisic Regressio (L-SLR) P( w) = P( w α) P( α) α w λ λ = ( πα) exp exp α α α λ w = ( π ) α exp λα α + α = λ exp( λ w ) Here we use he followig formula, b π x exp + ax x = exp( b a ). x a I... SLR-LAP W-sep: log Q( w) =< log P( y, w, α) > = Q( α) = log P( y w) +< log P ( w α) > + cos 0 Q( α) = { ylog σ + ( y) log( σ) } w Aw+ cos = E( w) ( w w) H ( w)( w w) + cos Q( w) ( w, S) where S H( w) a w is maximizer of E( w ). Usig he followig oaios: A = iag( α,, α ) : = E( w) = { ylog σ + ( y)log( σ) } w Aw. E y σ y σ = Aw w = σ w σ w = = { ( σ ) ( ) σ } = y y x Aw, = ( y σ ) x Aw = Xδ Aw

17 E w w σ = x A w = = ( σ ) σ xx A = = XBX ' A H( w) δ = [ y σ,, y σ ] : X = [ x,, x ] :. B = iag( σ ( σ ),, σ ( σ )) : A-sep: log Q( α) =< log P( ywα,, ) > = Q( w) =< log P ( w α) > + log P ( α) + cos 0 Q( w) 0 = ( α < w > logα ) logα + cos = = = ( α < w >+ logα) + cos = log Q( α ) Q( α ) = Q( α ) = Γ( α ; α, γ ) where = = γ w + s =, γ = α α = = w + s γ w, s are he h eleme a he h iagoal eleme of w a S, respecively. Fas covergece rule : α = w α w + s + α s = α w = α s α = γ =. αs w I... SLR wih variaioal approximaio (SLR-VAR)

18 From he variaioal approximaio o he logisic fucio(jaakkola a Jora 000), we have x ξ σ x σ ξ λ ξ x ξ ( ) ( )exp ( )( ) where λξ ( ) = σξ ( ) > 0 ( ξ> 0). Sice he likelihoo fucio ca be ξ rewrie as y P( y w) = σ ( σ ) he y y = σ (exp( z ) σ ) y = σ exp( z ( y )) z ξ { } P( y w) σξ ( )exp λξ ( )( z ξ ) z( y) h( y, w, ξ) where z = xw. Usig his formula, FE is lower boue as follows h( ywξ,, ) P0( w α) P0( α) FE( Q( w, α)) (, )log Q w α αw FE( Q( w, α), ξ). Q( w, α) FE ( Q( w, α), ξ ) is maximize wih respec o Q( w), Q( α ), ξ aleraely. W-sep: log Q( w) =< log h( y, w, ξ) + log P( w α) + log P( α) > = 0 0 Q( α) = log h( ywξ,, ) +< log P ( w α) > + cos 0 Q( α) z ξ = log σξ ( ) + λξ ( )( z ξ) z( y) w Aw+ cos = = { (y ) z λξ ( ) z} w Aw+ cos = { (y ) xw λξ ( ) wxx w} waw+ cos = = w A + x x w+ y x w+ cos ( ) ( 0.5) λξ = = ( ) w w ( w w ) = S + cos Thus Q( w) ( w, S) where

19 w = S( y 0.5) x = = ( A+ XΛX ) Xy = A X( Λ + X A X) Λ y : S = A+ λξ ( ) xx = A+ XΛ X : = y = [ y 0.5,, y 0.5] : X = [ x,, x ] : Λ= iag( λξ ( ),, λξ ( )) : -Sep: FE ( Q( w, α), ξ) =< log h( ywξ,, ) + log P0( w α) + log P0 ( α) > Q( w) Q( α) =< log h( ywξ,, ) > Q( w) Q( α) + cos ξ =< log σξ ( ) λξ ( )( z ξ) > Q( w) + cos = Takig parial erivaive of FE ( Q( w, α), ξ ), we have FE ξ σ ( ξ) = λ ( ξ)( < z > ξ) + λ( ξ) ξ σ( ξ) = σξ ( ) λ( ξ)( < z > ξ) + σξ ( ) = λ ξ < > ξ ( )( z ) Sice λ ( ξ ) 0for ξ > 0, FE = 0 ξ leas o ξ =< z > = x < ww > x = x ( ww + S) x. I.. RLR wih Laplace approximaio (RLR-LAP) A-sep:

20 log Q( α) =< log P( yw,, α) > Q( w) =< log P0 ( w α) > Q( w) + log P0( α) + cos = < ww> α + logα logα + cos = < ww> α + logα + cos Thus Q( α) =Γ( α ; α, γ ) where α =, γ = (5) w + s ( ) = α s = Fas upaig rule : Q( α) =Γ( α ; α, γ ) where α =, γ =. w = I.4. L-SLR wih Laplace approximaio (L-SLR-LAP) W-sep log Q( w) =< log P( y, w, α) > = Q( α) = log P( y w) +< log P ( w α) > + cos 0 Q( α) = { ylog σ + ( y) log( σ) } wvw+ cos = E( w) ( w w) H ( w)( w w) + cos Q( w) ( w, S) where S H( w) a w is maximizer of E( w ). Usig he followig oaios: V = iag( < α >,, < α > ) : = E( w) = { ylog σ + ( y)log( σ) } wvw. E y σ σ y = Vw w = σ w σ w = = { ( σ ) ( ) σ } = y y x Vw, = ( y σ ) x Vw = Xδ Vw

21 E w w σ = x V w = = ( σ ) σ xx V = = XBX ' V H( w) δ = [ y σ,, y σ ] : X = [ x,, x ] :. B = iag( σ ( σ ),, σ ( σ )) :