Why Linear? 4. Linear Discriminant Functions. Linear Discriminant Analysis. Theorem. Distance to a nonlinear function

Transcription

1 4. Lar Dscrmat Fuctos Alx M. Martz Hadouts Hadoutsfor forece ECE874, Why Lar? It s smpl ad tutv. Mmzg a crtro rror;.g. sampl rsk, trag rror, marg, tc. Ca b gralzd to fd o-lar dscrmat rgos. It s grally vry dffcult to calculat th dstac of a tstg sampl to a olar fucto. Lmtd umbr of trag sampls. No d to stmat class dstrbutos. Dstac to a olar fucto Lar Dscrmat Aalyss If w hav sampls corrspodg to two or mor classs, w prfr to slct thos faturs that bst dscrmat btw classs rathr tha thos that bst dscrb th data. hs wll, of cours, dpd o th classfr. Assum our classfr s Bays. hus, w wat to mmz th probablty of rror. W wll dvlop a mthod basd o scattr matrcs. From Muras & Nayar, 995. horm Lt th sampls of two classs b Normally dstrbutd R p, wth commo covarac matrx. h, th Bays rrors th p-dmsoal spac ad th o-dmsoal subspac gv by v ( - )/ ( - ), ar th sam; whr x s th Euclda orm of th vctor x. hat s, thr s o loss classfcato wh rducg from p dmsos to o. LDA PCA

2 Scattr matrcs ad sparablty crtra Wth-class scattr matrx: S W Btw-class scattr matrx: SB C N x j j xj j. j C j j. j Not that: S ˆ W S B. o formulat crtra for class sparablty, w d to covrt ths matrcs to umrcal valus: tr( S S ) ypcal combato of scattr matrcs ar: S, S } { S, S },{ S, ˆ }, ad{ S, ˆ}. l S tr S tr S l S { B W B W A soluto to LDA Aga, w wat to mmz th Bays rror. hrfor, w wat th projcto from Y to X that mmzs th rror: p X ˆ ( p) y b. p h gvalu dcomposto s th optmal trasformato: S. B S W Smultaous dagoalzato. Roald Fshr (890-96) Fshr was a mt scholar ad o of th grat sctst of th frst part of th 0 th ctury. Aftr graduatg from Cambrdg ad bg dd try to th Brtsh army for hs poor ysght, h workd as a statstca for sx yars bfor startg a farmg busss. Whl a farmr, h cotud hs gtcs ad statstcs rsarch. Durg ths tm, h dvlopd th wll-kow aalyss of varac (ANOVA) mthod. Aftr th war, Fshr fally movd to Rothamstd Exprmtal Stato. Amog hs may accomplshmts, Fshr vtd ANOVA, th tchqu of maxmum lklhood (ML), Fshr Iformato, th cocpt of suffccy, ad th mthod ow kow as Lar Dscrmat Aalyss(LDA). Durg World War II, th fld of ugcs suffrd a bg blow -- maly do to th Nazs us of t as a justfcato for som of thr actos. Fshr movd back to Rothamstd ad th to Cambrdg whrh rtrd. Fshr has b accrdtd to b o of th foudrs of modr statstcs ad o caot study pattr rcogto wthout coutrg svral of hs groud-brakg sghts. Yt as grat as a statstca that h was, h also bcom a major fgur gtcs. A classcal quot th Aals of Statstcs rads I occasoaly mt gtcsts who ask m whthr t s tru that th grat gtcst R.A. Fshr was also a mportat statstca." Exampl: Fac Rcogto

3 Lmtatos of LDA o prvt S W to bcom sgular, N>d. hr ar oly C- ozro gvctors. Noparamtrc LDA s dsg to solv th last problm(w l s ths latr th cours). PCA vrsus LDA I may applcatos th umbr of sampls s rlatvly small compard to th dmsoalty of th data. Ev for smpl PDFs, PCA ca outprform LDA (tstg data). Aga, ths lmts th umbr of faturs o ca us. PCA s usually a guarat, bcaus all w try to do s to mmz th rprstato rror. udrlyg but ukow PDFs Problms wth Mult-class Eg-basd Algorthms I gral, rsarchrs df algorthms whch ar optmal th -classs cas ad th xtd ths da (way of thkg) to th mult-class problm. hs may causd problms. hs s th cas for g-basd approachs whch us th da of scattr matrcs dfd abov. Lt s df th gral cas: M V M V hs s th sam as slctg thos gvctors v that maxmz: v M v. v M v Not that ths ca oly b achvd f M ad M agr. h xstc of soluto dpds o th agl btw th gvctors of M ad M. v s th th bass vctor of th soluto spac. w ar th gvctors of M. u ar th gvctors of M. 3

4 How to kow? Classfcato: h lar cas K r r cos j u j w. j j whr r < q ad q s th umbr of gvctors of M. h largr K s, th lss probabl that th rsults wll b corrct. Dcso Surfacs A lar dscrmat fucto ca b mathmatcally wrtt as: g x) w x w. ( o -class cas: wght vctor Dcd w f g( x) 0. w f g( x) 0. W ca also do that wth: w f w x w. o thrshold thrshold Dscrmat fucto = dstac h dscrmat fucto gvs a algbrac masur of th dstac. ak two vctors x ad x, both o th dcso boudary. h, wrt x as: x x p r w w Projcto of x oto g(x)=0. g( x) r. w Dstac from x to g(x)=0. Multcatgory cas wo straghtforward algorthms: Rduc th problm to C- -class problms. Costruct C(C-)/ lar dscrmat fuctos. A lar mach assgs x to: w f g ( x) g j ( x) j. h dcso boudars (btw two adjact aras) ar gv by: g ( x) g ( x). j 4

5 Larly Sparabl Our prvous algorthm s gral ad ca b appld v wh th classs ar ot larly sparabl. C classs ar larly sparabl ff for vry w thr xst a lar classfr (hyprpla) such that all th sampl of w l o ts postv sd (g (x)>0) ad all th sampl of ( j ) ar o th gatv sd of g (x). w j Lar Classfr If th classs (or trag data) ar larly sparabl, th thr xst a uqu g (x)>0. hs mas that a w sampl t, ca b classfd as w f g (t)>0. Altratvly (wh th data s ot larly sparabl), w ca us: w ( t w,..., wc, w0,..., w0 ) arg max g ( ). C t wghts thrsholds Lar Rgrsso It s smplr to start wth rgrsso. Rmmbr that rgrsso s a rlatd (but dffrt) problm to that of classfcato. W sarch that fucto g(x) whch bst trpolats a gv trag st S={(x,y ),,(x,y )}; whr x scalars. W wat to mmz: f p ad y ar w, x w x, ( x, y) y g ( x) y w, x 0. ad w hav assumd w o =0. If ough trag data s avalabl thr xst a uqu soluto: w X y. Wh thr s os (..,thr dos ot xst a g() for whch f()=0), w us last squars (LS). h (LS) rror fucto s gv by: E( w,( X, y)) Usg orm- LS: y g ( x ). y w X y w X. Now, dffrtatg wth rspct to w, w gt: y X w XX 0, w XX y X. If th vrs xst, th: w XX Xy. Wh thr s lss sampls tha dmsos (<p), thr xst may possbl solutos for w ll-codto problm. W d to mpos a rstrcto (or bas) to favor o soluto ovr th rst. hs s kow as rgularzato. 5

6 Rdg Rgrsso W r-formulat th problm as: m w y g (x ) m E w, X, y. w w Dffrtatg wth rspct to th paramtrs, w gt: w XX w w XX I p y X w XX I p Xy. prmal soluto. Not that for ay gv, w choos that soluto whch mmzs th orm of w. Not that w could hav also wrtt w as a fucto of th puts X: w X X w y X, X w y, X X y, X X I y, G I y. G=XX s th Gram matrx: G j x, x j. Dual soluto: w x. Gralzd Lar Dscrmat A look back at PCA Rmmbr that to comput th PCs of a dstrbuto wh <p, w also usd th ~ dual opto;.. Q X X. ( 3 ) vs. ( p 3 ). h two argumts (PCA ad LS) ar quvalt. o s ths, rmmbr that to mmz Ux usg LS, w hav E Ux x U Ux. h gvctor assocatd to th smallst gvalu of UU mmzs E. W ca rwrt our lar dscrmat as: d g (x) w0 w x. W ca ow xtd ths to a quadratc form: g (x) w0 w x j wj x x j. d d d Or to ay othr polyomal dscrmat fucto: g ( x) a y dˆ g (x) a y (x). g ( x) a a x a3 x y x x 6

7 rag h wght vctor a must b dtrmd from trag obsrvatos (sampls) of th world. h data must also b larly sparabl at last y. Krls W ca also addrss ths problm usg krls. If w cosdr x p (x) F q, w ca wrt f (x, y ) y w, (x). W hav alrady show that ths mpls: G j (x ), ( x j ). krl. W ow hav: g (x) G I yk, k (x ), ( x). Complxty h ovrall complxty of computg s O(3+q). Ad, to valuat g o a w sampl O(q). Krls ar vry usful, bcaus thy allow us to comput classfcato o a spac of q dmsos (grally, q>>p), wthout th d to spcfcally calculat th o-lar projcto x,z). Do ot b dcvd though. Mthods basd o krls ar computatoally xpsv ad slow. krl. Larly Sparabl: -class problm h goal s to fd th vctor a that corrctly classfs all sampls, y. Classfcato: a y 0 w a y 0 w No r ma l z a t o :t hr pl a c m to fa l lt h sampls of w by thr gatvs. h: a y 0,. No r mal z d :all sampls postvs 7

8 Gradt dsct procdur W df a crtro fucto J(a) that s mmzd wh a s a soluto of a y 0. W start wth som arbtrary valu ad th us th drcto of th gradt: J (a). Larg quato: a(k ) a(k ) (k ) J (a(k )). larg rat N wt o sd s c t Uss: a(k ) a(k ) H J. Algorthm:. Do a a H J. Utl H J (a) thrshold. 3. rtur a. W ow calculat th gradt: Prcptro J P (a) y O of th smpls crtra usd to mmz a y 0, s th Prcptro crtro. It oly uss thos fatur vctors that ar msclassfd: J P (a) ( a y ) hs crtro s vr gatv. Wh ths s zro, a s a soluto. Ad clud ths soluto our larg quato: a(k ) a(k ) (k ) y. Algorthm: Algorthm (k=0) Do k = mod (k+,) If y k s msclassfd by a th: a a y k Utl all pattrs ar proprly classfd 8

9 # of pattrs msclassfd J P (a) ( a y ) Covrgc horm (Prcpto Covrgc): If th trag sampls ar larly sparabl, th th squc of wght vctors gv by th Prcptro algorthm wll trmat at th soluto vctor. Prcptro: Batch varabl crmt W clud a marg b to th soluto: a (k )y k b k. Ad th st of proprly classfd sampls s gv by: a y k b. h slcto of b s grally dffcult. Wow Algorthm Rlaxato Procdur Co s d r st h pos t v a d g a t v wght vctors sparatly: a ad a. Corrctos o th postv wght ar mad oly wh o or mor trag pattrs w ar msclassfd. Corrctos o th gatv wght oly wh pattrs of w ar msclassfd. h Prcptro crtro dscrbd abov s by o mas th oly soluto. Aothr possblty would b: J q (a) (a y ). Its gradt s cotuous (whras that of J p s ot). Ufortuatly, th fucto s too smooth. Covrgc s usually at a = 0. 9

10 Aothr problm wth J q s that t s domatd by th largst trag vctors. Istad, w usually us th followg: (a y b) J r (a) y margal a y b J r y. y Its gradt s: Ad th larg (updat) quato s: a y b a(k ) a(k ) ( k ) y y. Mmum Squar-Error Procdur W wll ow buld a clos-form soluto that uss all sampls y smultaously. For that w shall try to mak a y k b. A soluto s gv by th followg st of lar quatos: y0 y y0 y y 0 y yd a0 b y d a b yd ad b h gradt s: (a y b) J r (a ) y I matrx form: Ya b. Ufortuatly, Y s rctagular, ad thr ar mor quatos tha ukows => a s, thrfor, ovrdtrmd. Aga, w d to mmz a rror;.g.: rror Ya b. form whch w ca df th followg crtro: J S (a) Ya b (a y b ). A soluto s ow gv by: J S (a y b ) y Y ( Ya b). J q (a) (a y ). a (Y Y) Y b Y*b. Now, w wat to mmz ths: Y (Ya b) 0 Y Ya Y b 0 Y Ya Y b Why s ths w quato mportat? Bcaus Y Y s a squar matrx. MSE psudovrs If Y s squar ad osgular => th psudovrs = vrs. A soluto ca always b foud wth: Y * lm ( Y Y I) Y. 0 MSE = LDA. 0

11 A smpl xampl w : (,), (,0) w : (3,), (,3) wo problms of MSE:. Y Y ca b sgular.. Y s grally a vry larg matrx. a x 0 x 0 Y 3 3 LMS (last-ma-squar) procdur 5 / 4 3 / 3 / 4 7 / Y * / / 6 / / 6 0 / 3 0 / 3 W ca solv ths by mmzg th abov dfd crtro: J S (a) Ya b. h updat quato would b: a(k ) a(k ) (k )Y (Ya(k ) b). Usually howvr, w cosdr th sampl squtally: a(k ) a(k ) (k )(bk y k a (k ))y k. Advatag: lss mmory rqurd. It ca b show that wth (k ) () / k ths algorthm covrgs. Ufortuatly, th soluto ds ot gv a sparatg vctor (v f o xsts). So far, so good h Ho-Kashyap procdur h Prcptro fds th sparatg vctor wh th data s larly sparabl, but dos ot covrg o o-sparabl sts. Covrgc s guaratd MSE & LMS, but th rsultg vctor ds ot b th corrct o. W wll ow df aothr algorthm whch attmpts to addrss ths problm. h MSE mmzs: Ya b. hr xst a b for whch th MSE ylds a corrct soluto. Howvr, b s ukow. h Ho-Kashyap procdur sarchs for both, a ad b. h quato to mmz s: J S (a, b) Ya b.

12 Ad ts gradts ar: a J S Y Ya b b J S Ya b Bcaus a Y *b, o oly ds to mmzd th crtro wth rspct to b. Howvr, w also d to guarat that b>0. W ca do that wth a updat quato whch s always postv: If th sampls ar larly sparabl, covrgc s guaratd. If thy ar ot, th Ya(k ) b(k ) 0 ad th algorthm stops. hs ca b usd as a proof that th sampls ar ot sparabl. b() 0, b(k ) b( k ) ( k ) b J S b J S. Support Vctor Machs As w dd bfor th goal s to rprst th data a hgh-dmsoal spac whr th data s larly sparabl. W us a olar mappg (); as for xampl a polyomal or a Gaussa mappg fucto. Optoal homwork: show that ay datast that blogs to two classs ca b sparatd by a hyprpla f th data s rprstd a suffctly larg spac. As usual w hav g ( y ) a y ad w wat to fd th sparatg hyprpla, zk g (y k ), wth largst marg b: zk g ( y k ) a b. h goal s to fd th wght vctor a that maxmzs b. hs s th sam as: fd th sparatg hyprpla that s farthst from th most dffcult pattrs to classfd. A SVM algorthm Rcalculat a usg th Prcptro algorthm ad th worst-classfd pattrs: a(k ) a( k ) (k ) y. worst y h rsultg a s calld th support vctor.

13 Support Vctors W wat to fd thos sampls that ar closst to th hyprpla that dvds our fatur spac to two rgos (+ ad -). W ca formulat ths as follows: r a y a0, a marg h sampls that ar closst to th marg ar calld th support vctors. Our goal s to maxmz. o solv ths problm w d a rgularzg trm. As w dd bfor, w wll mmz w. Formally, a, subjct to r a y a0. m whr ={,, } (# of sampls), ad r= f th th sampl blogs to class ad f th sampl blogs to th scod class. Soft Marg SVM Lagrag Multplrs W ca solv ths quadratc optmzato problm usg Lagrag multplrs. a r a y a0 h goal s to mmz wth rspct to w ad maxmz wth rspct to. Our prvous SVM algorthms may ot prform adquatly wh th data s olarly sparabl. A soft marg SVM allows for th msclassfcato of a fw sampls whl stll maxmzg th marg for th rst. W ca modl th rror of ach sampl as follows, r a y a0. Rcostructo rror (as PCA). Multclass Classfcato K s l r scostructo Ufortuatly, thr s o gral way to achv multclass classfcato. A larg mach: W start wth a st of c- qualts: g (x) a y augmtd vctor z [a, a,..., a c ] x s assgd to w f g (x) g j (x) j. Whch s th sam as: a y k aj y k, j. a, a,..., a c a y k aj y k, 0, j,..., c. # of classs y y y 0 0 y 0, 3,..., c 0 y 0 0 y h goal s to solv: z j 0. 3

14 Gralzato of MSE h oly straghtforward soluto s to cosdr a st of c -class problms: a y y st of vctors class a y y A [a, a,..., a c ] B [b, b,..., b c ] A Y *B. 4