Chapter 3: Maximum-Likelihood & Bayesian Parameter Estimation (part 1)

Transcription

1 Aoucemes Reags o E-reserves Proec roosal ue oay Parameer Esmao Bomercs CSE 9-a Lecure 6 CSE9a Fall 6 CSE9a Fall 6 Paer Classfcao Chaer 3: Mamum-Lelhoo & Bayesa Parameer Esmao ar All maerals hese sles were ae from Paer Classfcao e by R. O. ua, P. E. Har a. G. Sor, Joh Wley & Sos, wh he ermsso of he auhors a he ublsher Irouco Mamum-Lelhoo Esmao Eamle of a Secfc Case The Gaussa Case: uow a Bas Irouco aa avalably a Bayesa framewor We coul esg a omal classfer f we ew: P rors P class-cooal eses Uforuaely, we rarely have hs comlee formao! A ror formao abou he roblem Normaly of P P ~ N, Σ Characerze by arameers Esmao echques 5 esg a classfer from a rag samle No roblem wh ror esmao Samles are ofe oo small for class-cooal esmao large meso of feaure sace! Mamum-Lelhoo ML a he Bayesa esmaos Resuls are early ecal, bu he aroaches are ffere Paer Classfcao, Chaer 3 Paer Classfcao, Chaer 3

2 Parameers ML esmao are fe bu uow! Bes arameers are obae by mamzg he robably of obag he samles observe Bayesa mehos vew he arameers as raom varables havg some ow srbuo I eher aroach, we use P for our classfcao rule! 6 Mamum-Lelhoo Esmao Has goo covergece roeres as he samle sze creases Smler ha ay oher alerave echques Geeral rcle Assume we have c classes a P ~ N, Σ P P, where: 7, Σ,,...,,,cov m,... Paer Classfcao, Chaer 3 Paer Classfcao, Chaer 3 Use he formao rove by he rag samles o esmae,,, c, each,,, c s assocae wh each caegory 8 9 Suose ha coas samles,,,, P P F P s calle he lelhoo of w.r.. he se of samles ML esmae of s, by efo he value ha ˆ mamzes P I s he value of ha bes agrees wh he acually observe rag samle Paer Classfcao, Chaer 3 Paer Classfcao, Chaer 3 Omal esmao Le,,, a le be he grae oeraor We efe l as he log-lelhoo fuco l l P New roblem saeme: eerme ha mamzes he log-lelhoo,,..., Se of ecessary coos for a omum s: l lp l ˆ argma l Paer Classfcao, Chaer 3 Paer Classfcao, Chaer 3

3 Eamle of a secfc case: uow, Σ ow P ~ N, Σ Samles are raw from a mulvarae ormal oulao Mullyg by Σ a rearragg, we oba: ˆ 3 l P l a l P [ π Σ ] Jus he arhmec average of he samles of he rag samles! herefore: The ML esmae for mus sasfy: Σ ˆ Cocluso: If P,,, c s suose o be Gaussa a - mesoal feaure sace; he we ca esmae he vecor,,, c a erform a omal classfcao! Paer Classfcao, Chaer 3 Paer Classfcao, Chaer 3 ML Esmao: Gaussa Case: uow a,, l l P l π l P l l P Summao: ˆ ˆ ˆ ˆ Combg a, oe obas: ˆ ˆ ˆ ; ˆ 5 Paer Classfcao, Chaer 3 Paer Classfcao, Chaer Bas ML esmae for s base E Σ A elemeary ubase esmaor for Σ s: C ˆ - 3 Samle covarace mar Ae: ML Problem Saeme Le {,,, } P,, Π, P ; Our goal s o eerme ˆ value of ha maes hs samle he mos rereseave! Paer Classfcao, Chaer 3 Paer Classfcao, Chaer 3 3

4 Bayesa Parameer Esmao ar 8 Bayesa Esmao 9 Bayesa Esmao BE Bayesa Parameer Esmao: Gaussa Case Bayesa Parameer Esmao: Geeral Esmao Problems of mesoaly Comuaoal Comley Comoe Aalyss a scrmas He Marov Moels I MLE was suose f I BE s a raom varable The comuao of oseror robables P ha s use for classfcao les a he hear of Bayesa classfcao Gve he samle, Bayes formula ca be wre, P P, c, P We assume ha - Samles rove fo abou class oly, where {,, c } So ow wha o we o??? Well, he oly erm we o ow o he rgh-se of - P P.e., samles eerme he ror o I P, c, P, P P, c, P, P s, he class cooal esy, bu hs volves a arameer ha s a raom varable. Goal: comue, If we ew we woul be oe! Bu we o ow. Bayesa Parameer Esmao: Gaussa Case 3 We o ow ha - has a ow ror Se I: Esmae usg he a-oseror esy P - a we have observe samles. So we ca re-wre he cc as,, The uvarae case: P s he oly uow arameer ~ N, a are ow! ~ N, 3

5 5 So ow we mus calculae Reroucg esy s fou as where N, ~ ˆ a α 5 6 Bayesa Parameer Esmao: Gaussa Case Se II: remas o be comue! So he esre cc ca be wre as s Gaussa, ~ N 7 Bayesa Parameer Esmao: Gaussa Case Se III: We o hs for each class a combe P, wh P alog wh Bayes rule o ge [ ] [ ].P, P Ma, P Ma 8 Bayesa Parameer Esmao: Geeral Theory P comuao ca be ale o ay suao whch he uow esy ca be arameerze. The basc assumos are: - The form of P s assume ow, bu he value of s o - Our owlege abou s coae a ow ror esy P - The res of our owlege s coae a se of raom varables,,, ha follows P 9 The basc roblem s: Se I: Comue he oseror esy P Se II: erve P Se III: Comue, Usg Bayes formula, we have: A by a eeece assumo:

6 Why o We Always Acqure More Feaures? 3 Problems of mesoaly 3 Coser case of wo classes mulvarae ormal wh he same covarace: 7 Perror where : r π u e r / u Σ lm Perror r 3 If feaures are eee he: Σ ag r,,..., Mos useful feaures are he oes for whch he fferece bewee he meas s large relave o he saar evao I has frequely bee observe racce ha, beyo a cera o, he cluso of aoal feaures leas o worse raher ha beer erformace: we have he wrog moel! 7 6