//6 S 688 Pr Rcogiio Lir Modls for lssificio Ø Probbilisic griv modls Ø Probbilisic discrimiiv modls Probbilisic Griv Modls Ø W o ur o robbilisic roch o clssificio Ø W ll s ho modls ih lir dcisio boudris ris from siml ssumios bou h disribuio of h d
//6 Griv Aroch Ø Solv h ifrc roblm of simig h clsscodiiol dsiis ( for ch clss Ø Ifr h rior clss robbiliis Ø Us Bs horm o fid h clss osrior robbiliis: ( ( hr! ( ( ( Ø Us dcisio hor o drmi clss mmbrshi for ch iu Probbilisic Griv Modls o clss cs: ( ( ( ( ( ( ( σ l ( ( ( ( ( σ ( logisic sigmoid fucio
//6 3 Probbilisic Griv Modls > clsss: l σ sofm fucio Probbilisic Griv Modls D / / Ls ssum h clss-codiiol dsiis r Gussi ih h sm covric mri: o clss cs firs. W c sho h folloig rsul: l σ
//6 4 Probbilisic Griv Modls l l l l l l l σ ] [ / / D Probbilisic Griv Modls l l l
//6 5 Probbilisic Griv Modls W hv sho: Dcisio boudr: l σ 5..5 Probbilisic Griv Modls
//6 6 Probbilisic Griv Modls > clsss: W c sho h folloig rsul: l l Probbilisic Griv Modls h ccllio of h qudric rms is du o h ssumio of shrd covrics. If llo ch clss codiiol dsi o hv is o covric mri h ccllios o logr occur d obi qudric fucio of i.. qudric discrimi.
//6 Probbilisic Griv Modls Mimum lilihood soluio W hv rmric fuciol form for h clss-codiiol dsiis: ( ( ( D/ / ( W c sim h rmrs d h rior clss robbiliis usig mimum lilihood. Ø o clss cs ih shrd covric mri. Ø riig d: { } dos Priors : ( ( ( clss dos clss 7
//6 Mimum lilihood soluio { } dos Priors : ( ( ( clss dos clss For d oi from clss hv d hrfor For d oi from clss hv d hrfor ( ( ( ( ( { } Mimum lilihood soluio dos clss dos clss : ( ( ( ( ( ( ( ( ( ( ( ( Priors Assumig obsrvios r dr iddl c ri h lilihood fucio s follos: ( " # $ ( " # [ ( ] $ [ ] % [" ( #] $ ( % " # [ ] % (! 8
//6 9 Mimum lilihood soluio " # " # [ ] $ % " # [ ] % W o fid h vlus of h rmrs h mimiz h lilihood fucio i.. fi modl h bs dscribs h obsrvd d. As usul cosidr h log of h lilihood: [ ] l l l l l Mimum lilihood soluio W firs mimiz h log lilihood ih rsc o. h rms h dd o r [ ] l [ ] l l [ ] l l l l l
//6 Mimum lilihood soluio hus h mimum lilihood sim of is h frcio of ois i clss h rsul c b grlizd o h muliclss cs: h mimum lilihood sim of is giv b h frcio of ois i h riig s h blog o ML Mimum lilihood soluio W o mimiz h log lilihood ih rsc o. h rms h dd o r [ ] l l l l l [ ] l ] [ / / D [ ] [ ] [ ] [ ] cos l
//6 Mimum lilihood soluio hus h mimum lilihood sim of is h sml m of ll h iu vcors ssigd o clss B mimizig h log lilihood ih rsc o obi similr rsul for ( Mimum lilihood soluio Mimizig h log lilihood ih rsc o h mimum lilihood sim ML ML S S S S obi ( ( Ø hus: h mimum lilihood sim of h covric is giv b h ighd vrg of h sml covric mrics ssocid ih ch of h clsss. Ø his rsuls d o clsss.
//6 Probbilisic Discrimiiv Modls o-clss cs: σ ( Muliclss cs: ( Discrimiiv roch: us h fuciol form of h grlizd lir modl for h osrior robbiliis d drmi is rmrs dircl usig mimum lilihood. Probbilisic Discrimiiv Modls Advgs: Ø Fr rmrs o b drmid Ø Imrovd rdiciv rformc scill h h clss-codiiol dsi ssumios giv oor roimio of h ru disribuios.
//6 Probbilisic Discrimiiv Modls o-clss cs: ( ( σ ( ( I h rmiolog of sisics his modl is o s logisic rgrssio. Assumig sim? M R ho m rmrs do d o M Probbilisic Discrimiiv Modls Ho m rmrs did sim o fi Gussi clss-codiiol dsiis (griv roch? ( m vcors M M M M ol M M M M M O M 3
//6 Logisic Rgrssio ( ( σ W us mimum lilihood o drmi h rmrs of h logisic rgrssio modl. { }! dos clss dos clss W o fid h vlus of h mimiz h osrior robbiliis ssocid o h obsrvd d Lilihood fucio : P( L " # P ( # ( " # ( # L Logisic Rgrssio ( ( σ ( ( P P W cosidr h giv logrihm of h lilihood: l L l ( ( ( ( l ( ( l( ( rg mi 4
//6 5 Logisic Rgrssio W comu h driviv of h rror fucio ih rsc o (grdi: [cross ro rror fucio] W d o comu h driviv of h logisic sigmoid fucio: σ l l " " # " " $ $ $ $ $ $ $ $ $ % & ' ( * # $# Logisic Rgrssio σ l l
//6 Logisic Rgrssio ( Ø h grdi of givs h dircio of h ss icrs of. W d o miimiz. hus d o ud so h mov log h oosi dircio of h grdi: his chiqu is clld grdi dsc Ø I c b sho h is cocv fucio of. hus i hs uiqu miimum. Ø A ffici iriv chiqu iss o fid h oiml rmrs (o-rhso oimizio. Bch vs. o-li lrig ( Ø h comuio of h bov grdi rquirs h rocssig of h ir riig s (bch chiqu Ø If h d s is lrg h bov chiqu c b cosl; Ø For rl im licios i hich d bcom vilbl s coiuous srms m o ud h rmrs s d ois r rsd o us (o-li chiqu. 6
//6 O-li lrig Ø Afr h rsio of ch d oi comu h coribuio of h d oi o h grdi (sochsic grdi: Ø h o-li udig rul for h rmrs bcoms: η " > is clld lrig r. ( η( I's vlu ds o b chos crfull o sur covrgc Muliclss Logisic Rgrssio Muliclss cs: ( W us mimum lilihood o drmi h rmrs of h logisic rgrssio modl. { }! (!! dos clss W o fid h vlus of! h mimiz h osrior robbiliis ssocid o h obsrvd d Lilihood fucio : L! P( "" "" 7
//6 Muliclss Logisic Rgrssio L ( ( ( W cosidr h giv logrihm of h lilihood: ( l L( l ( rg mi ( Muliclss Logisic Rgrssio ( l ( ( W comu h grdi of h rror fucio ih rsc o o of h rmr vcors: l ( ( 8
//6 Muliclss Logisic Rgrssio ( hus d o comu h drivivs of h sofm fucio: " " # $ " " # $ $ % ( '# * & ( ( Muliclss Logisic Rgrssio ( hus d o comu h drivivs of h sofm fucio: ( ( for " # # # # $ % & ( ' $ % * 9
//6 Muliclss Logisic Rgrssio omc rssio: for h idi mri r h lms of I hr I Muliclss Logisic Rgrssio I I l I
//6 Muliclss Logisic Rgrssio ( ( Ø I c b sho h is cocv fucio of. hus i hs uiqu miimum. Ø For bch soluio c us h o-rhso oimizio chiqu. Ø O-li soluio (sochsic grdi dsc: η η(