Classification. Linear Classification. What is a Linear Disciminant? Representing Classes. Decision Boundaries. What can be expressed?

Transcription

1 Classfcao Lar Classfcao Ro Parr CPS 7 Survsd larg framork Faurs ca b ayhg args ar dscr classs: Saf mushrooms vs. osoous Malga vs. bg Good crd rsk vs. bad Ca ra classs as umbrs? Sgl class? Mul class? Wh co adad from Adr Ng, Ls Goor, ad om Drch Fgurs from book coursy of Chrs Bsho ad Chrs Bsho Rrsg Classs Irr as h robably ha h h lm s a arcular class Classs usually dso For mulclass, s a vcor [] f h lm s class, OW Noao: For covc, ll somms rfr o h ra varabls, rahr ha h faurs as s hrough h ls of our faurs, φ Wha s a Lar Dscma? Smls kd of classfr, a lar hrshold u LU: f L y ohrs θ W somms assum, so y A lar dscrma s a - dmsoal hyrla s orhogoal o hs W ll look a hr algorhms, all of hch lar lar dcso boudars: Drcly lar h LU: Usg Las Ma Squar LMS algorhm Lar h codoal dsrbuo: Logsc rgrsso Lar h o dsrbuo: Lar dscrma aalyss LDA Dcso Boudars A classfr ca b vd as arog h u sac or faur sac X o dcso rgos A lar hrshold u alays roducs a lar dcso boudary. A s of os ha ca b sarad by a lar dcso boudary s larly sarabl. Wha ca b rssd? Eamls of hgs ha ca b rssd Assum boola/ faurs Coucos: ^ 3^ 4 : ^ 3^ 4 : a-las-m-of- a-las--of,, Eamls of hgs ha cao b rssd: No-rval dsucos: ^ 3 3^ 4 Eclusv-Or ^ ^

2 No-larly sarabl aml Mulclass k classs Ok o vs. o classfrs Esv May o b coss k- o vs. rs classfrs Lss sv Sll may o b coss K lar fucos Assg o class f > for all Gvs cov, sgly cocd dcso rgos Ho o ck h lar fucos? Why o us rgrsso? Rgrsso mmzs sum of squard rrors o arg fuco Gvs srog fluc o oulrs h Nural Sory Par I Nc o usfy mach larg /aur Naïv rosco orks badly Nural modl bologcally lausbl Sgl uro, lar hrshold u rcro Logr ra o hs lar Prcro Prcro Larg od/ uro f Y W ar gv a s of us s a s of arg ouus boola {-,} s our s of ghs ouu of rcro Prcro_rror, -, Goal: Pck o omz: f s a sml s fuco sg m msclassf d rcro_ rror,

3 msclassfd &5/,9#: : α!! "" # # # " "$ " %&#! Prcro Larg Prors LU Prors Good s: If hr ss a s of ghs ha ll corrcly classfy vry aml, h rcrolarg rul ll fd Dos o dd o s sz Bad s: Prcrosca rrs oly a small class of fucos, larly sarabl, fucos May osclla f o sarabl No obvous gralzao for mulclass Logsc Rgrsso I logsc rgrsso, lar h codoal dsrbuo P L b our sma of P, hr s a vcor of adusabl aramrs. Assum hr ar o classs, ad ad hs s quval o log Why hs form? O raso: rasforms a lar fuco h rag -, o b osv ad sum o so ha ca rrs a robably IOW, h log odds of class s a lar fuco of Cosrucg a Larg Algorhm Fd h robably dsrbuo h ha s mos lkly, gv h daa. P X h P h arg ma P h X arg ma by Bays Rul h h P X arg ma P X h P h bcaus PX dos dd o h h arg ma P X h f assum Ph s uform h arg ma log P X h bcaus log s mooo h h lklhood fuco vs PX h as a fuco of h aramrs h modl. I hs cas, our aramrs ar h ghs,. LX PX h h log lklhood s a commoly usd obcv fuco for larg algorhms. I s dod lx h ha mamzs h lklhood of h rag daa s calld h mamum lklhood smaor Log Lklhood for Codoal Probably Esmaors W ca rss h log lklhood a comac from calld h cross-roy ak a aml, f y, h log lklhood s log- f y, h log lklhood s log θ hs o ar muually clusv, so ca comb hm o g: l [ ] log, log P, log h goal of our larg algorhm ll b o fd o mamz: J l X,

4 Comug h Grad, J θ l [ ] log log l [ ] [ ] [ ] Grad co. Aohr ay of rg h logsc rgrsso fuco s: So g: Grad co. h grad of h loglklhood for a sgl o s: h ovrall grad s:, l [ ] [ ] J Comar /rco rul! Summary of Logsc Rgrsso Lars h Codoal Probably Dsrbuo P No closd form soluo Vry sml rsso for grad Solv by local sarch: Bgs h al gh vcor. Grad asc o mamz obcv fuco. Obcv fuco s h log lklhoodof h daa Algorhm sks h robably dsrbuo P ha s mos lkly gv h daa. May b do ol or bach Ca b usd h acclrao mhods No- Rahso, c. Wha W Alrady Ko Lar hrshold U LU rs o dscovr a lar fuco faur sac ha saras osv ad gav amls Logsc Rgrsso Uss rgrsso o sma h fuco log Dsy Esmao Basc usurvsd larg chqu Dscussd hr co of classfcao Ida: Esma o robably of faurs ad class labls

5 Dscr Cas Suos ko PX X Ho do g hs? Mamum lklhood sma coms from coug rlav frqucy Broul dsrbuo Bg o y Assumg: Bary loss fuco Chocs:, Favor h P > P Us dfo of codoal robably: W s a Wha s our guss for? P... P... P... So, ar do??? Ho may aramrs dd for o? Is hs raccal? Smlfcao Naïv Bays: P X... X P X Q: Ho s hs mor raccal? Naïv Bays Aco Sam flrg X X: Sam rlad faurs : Sam labl Comb Bays Rul /Naïv Bays: P X... X P P X... X P X... X P X P P X... X hgs o o: Do orry abou PX X? Ifluc of P? Is Naïv Bays Rasoabl? Ar faurs corrlad h classs? Ho ould hur us f hy r? Lar Dscrma Aalyss I LDA, lar h dsrbuo P W assum ha s couous W assum P s dsrbud accordg o a mulvara ormal dsrbuo ad P s a dscr dsrbuo Mor o hs h dscuss Baysa orks

6 Esmag h MVG aramrs Gv a s of daa os {,, N }, h mamum lklhood smas for h aramrs of h MVG ar: ˆ µ N Σˆ N ˆ µ ˆ µ Pug all oghr LDA Also calld Gaussa Dscrma Aalyss Hr Broull Νµ, Σ Νµ, Σ Wrg hs ou, g: / µ Σ / Σ µ π / / µ Σ Σ µ π Pckg A Class W aga us Bays rul: MVG codoal faur robably P X P P X P X Posror labl robably Pror class robably Pror faur robably gord h Bauy of Homoscdascy Rcall assumd Σsam for all classs Wh s Py >Py??? / / µ Σ µ y > π Σ / / µ Σ µ y π Σ µ Σ µ > µ Σ k µ Lar!!! 3 4 Eaml Homoscdasc LDA Dscusso For mulclass, hs gvs cov dcso boudars Sasfs dsdraa for mulclass dcso boudars Ho ralsc s hs? Wha do gv u? h dcso boudary s a y.5

7 Hroscdasc Dsrbuos Comarg LU, LR, LDA Bg dba abou h rlav mrs of drc classfrs lk LU vrsus codoal modls lk LR vrsus grav modls lk LDA '*,-. *-,/ 3 4,5,- 67, 89438: LDA vs LR Issus Wha s h rlaosh? I LDA, urs ou h ca b rssd as a logsc fuco hr h ghs ar som fuco of µ, µ, ad Σ! Bu, h covrs s NO ru. If s a logsc fuco, ha dos o mly s MVG LDA maks srogr modlg assumos ha LR h hs modlg assumos ar corrc, LDA ll rform br LDA s asymocally ffc: h lm of vry larg rag ss, hr s o algorhm ha s srcly br ha LDA hovr, h hs assumos ar corrc, LR s mor robus akr assumos, mor robus o dvaos from modlg assumos f h daa ar o-gaussa, h h lm, logsc ourforms LDA For hs raso, LR s a mor commoly usd algorhm Sascal ffccy: f h grav modl s corrc, h usually gvs br accuracy, scally for small rag ss. Comuaoal ffccy:grav modls ycally ar h ass o comu. I LDA, smad h aramrs drcly, o d for grad asc Robusss o chagg loss fuco: Boh grav ad codoal modls allo h loss fuco o chag hou rsmag h modl. hs s o ru for drc LU mhods Robusss o modl assumos: h grav modl usually rforms oorly h h assumos ar volad. Robusss o mssg valus ad os: I may alcaos, som of h faurs may b mssg or corrud for som rag amls. Grav modls rovd br ays of hadlg hs ha o-grav modls.