Bayesian Decision Theory

Transcription

1 No.4 Bayesan Decson Theory Hu Jang Deartment of Electrcal Engneerng and Comuter Scence Lassonde School of Engneerng York Unversty, Toronto, Canada

2 Outlne attern Classfcaton roblems Bayesan Decson Theory How to make the otmal decson? Generatve models: Maxmum a osteror MA decson rule: leadng to mnmum error rate classfcaton. Model estmaton: maxmum lkelhood, Bayesan learnng, dscrmnatve tranng, etc.

3 attern Classfcaton roblem Gven a fxed set of fnte number of classes:, 2,, N. We have an unknown attern/object wthout ts class dentty. BUT we can measure observe some features about : s called feature or observaton of the attern. can be scalar or vector or vector sequence can be contnuous or dscrete attern classfcaton roblem:! Determne the class dentty for any a attern based on ts observaton or feature. Fundamental ssues n attern classfcaton How to make an otmal classfcaton? In what sense s t otmal?

4 Examles of attern classfcatoni Seech recognton: attern: voce soken by a human beng Classes: language words/sentences used by the seaker Features: seech sgnal characterstcs measured by a mcrohone " a sequence of feature vectors Each vector: contnuous, hgh-dmensonal, real-valued numbers Natural language understandng: attern: wrtten or soken languages of human Classes: all ossble semantc meanngs or ntentons Features: the used words or word-sequences sentences Dscrete, scalars or vector

5 Examles of attern classfcatonii Image understandng: attern: gven mages Classes: all known object categores Features: color or gray scales n all xels Contnuous, multle vectors/matrx Examles: face recognton, OCR otcal character recognton. Gene fndng n bonformatcs: attern: a newly sequenced DNA sequence Classes: all known genes Features: all nucleotdes n the sequence Dscrete; 4 tyes adenne, guanne, cytosne, thymne roten classfcaton n bonformatcs: attern: roten rmary -D sequence Classes: all known roten famles or domans Features: all amno acds n the sequence: dscrete; 20 tyes

6 Bayesan Decson TheoryI Bayesan decson theory s a fundamental statstcal aroach to all attern classfcaton roblems. attern classfcaton roblem s osed n robablstc terms. Observaton s vewed as random varables vectors, Class d s treated as a dscrete random varable, whch could take values, 2,, N. Therefore, we are nterested n the jont robablty dstrbuton of and whch contans all nfo about and., If all the relevant robablty values and denstes are known n the roblem we have comlete knowledge of the roblem, Bayesan decson theory leads to the otmal classfcaton Otmal " Guarantee mnmum average classfcaton error The mnmum classfcaton error s called the Bayes error.

7 Bayesan Decson TheoryII In attern classfcaton, all relevant robabltes mean: ror robabltes of each class,2,,n: how lkely any a attern comes from each class before observng any features! ror knowledge from revous exerence N All rors sum to one: Class-condtonal robablty densty functons of the observed feature,,,2,,n: how the feature dstrbutes for all atterns belongng to one class. If s contnuous, s a contnuous.d.f. dstrbuton For every class : d If s dscrete, s dscrete robablty mass functon.m.f. dstrbuton. For every class :

8 Examle of class-condtonal.d.f. x2 x x.8 Fgure from Duda et. al., attern classfcaton John Wley & Sons, Inc.

9 Bayes Decson Rule: Maxmum a osteror MA I If not observe any feature of an ncomng unknown attern, classfy t based on ror knowledge only arg max Roughly guess t as the class wth largest ror robablty If observe some features of the unknown atter, we can convert the ror robablty nto a osteror robablty based on the Bayes theorem: osteror ror lkelhood evdence

10 Bayes decson rule: Maxmum a osteror MA II Where ror : robablty of recevng a attern from class before observng anythng. ror knowledge Lkelhood : robablty of observng feature f assume comes from a attern n class. f assume s gven, treat t as a functon of, t s called lkelhood functon osteror : robablty of gettng a attern from class after observng ts features as. Evdence x: a scalar factor to guarantee osteror robabltes sum to one regardng.

11 Bayes decson rule: Maxmum a osteror MA II If we observe some features of an unknown attern, the observaton can convert the ror nto osteror. Intutvely, we can class the attern based on the osteror robabltes, resultng n the maxmum a osteror MA decson rule, also called Bayes decson rule. For an unknown attern, after observng some features, we classfy t nto the class wth the largest osteror robablty: arg max arg max arg max

12 The MA decson rule s otmal I How well the MA decson rule behaves?? Otmalty: assume we have comlete knowledge, ncludng and,2,,n, the MA decson rule s otmal to classfy atterns, whch means t wll acheve the lowest average classfcaton error rate. roof of otmalty of the MA rule: Gven a attern, f ts true class d s, but we classfy t as, then the classfcaton error s counted as l : l 0 whch s also known as 0- loss functon.

13 The MA decson rule s otmal II l R N roof of otmalty of the MA rule cont For a attern, after observng, the osteror s the robablty that the true class d of s. Thus the exected average classfcaton error assocated wth classfyng as s calculated: The otmal classfcaton s to mnmze the above average classfcaton error,.e., f observng, we classfy as to mnmze R! maxmze! the MA decson rule s otmal, whch acheves the mnmum average average error rate. The mnmum error s called Bayes error.

14 The MA decson rule A general decson rule s a mang functon, gven any an observaton, outut a class d : " If we totally have N classes, a decson rule wll artton the entre feature sace of nto N dfferent regons, O, O2,, ON. If s located n the regon O, we classfy t as class. Each regon O could consst of many contguous areas. The MA decson rule the Bayes decson rule s otmal among all ossble decson rules n terms of mnmzng average classfcaton errors condtonal on that we have comlete and recse knowledge about the underlyng roblem. Feature sace Class Class 2 Class N

15 The MA decson rule examle O2 O O2 O Fgure from Duda et. al., attern classfcaton John Wley & Sons, Inc.

16 Classfcaton Error robablty of a decson rule Assume N-class roblem, any a decson rule artton the feature sace nto N regons, O, O2,, ON. denotes the robablty of the observaton of a attern ts true class d s j falls n the regon O. The overall classfcaton error robablty of the decson rule s:, r j O N O N N O O correct error d r, r r r

17 Examle of Error robablty n 2-class case Error 2 " Error " 2 Fgure from Duda et. al., attern classfcaton John Wley & Sons, Inc.

18 Bayes Error Bayes error: error robablty of the Bayes MA decson rule. Snce Bayes decson rule guarantees the mnmum error, the Bayes error s the lower bound of all ossble error robabltes. It s dffcult to calculate the Bayes error, even for the very smle cases because of dscontnuous nature of the decson regons n the ntegral, esecally n hgh dmensons. Some aroxmaton methods to estmate an uer bound. Chernoff bound Bhattacharyya bound Evaluate on an ndeendent test set.

19 Examle: Bayes decson for ndeendent bnary features Bayes decson rule the MA rule s also alcable when feature s dscrete. A smle case Bnomal model: 2-class, 2, feature vector s d-dmensonal vector, whose comonents are bnary-valued and condtonally ndeendent. d x x d x x t d q q x q x d x x x x r and r 0,,,,

20 Examle: Bayes decson for ndeendent bnary features The MA decson rule:. otherwse, classfy to 0, g If ln ln ln ln ln ln we have the decson functon : Equvalently, otherwse, f classfy to λ λ λ λ d d d q q q x q x q x g

21 Mssng features/data I If we know the full robablty structure of a roblem, we can construct the otmal Bayes decson rule. In some ractcal stuatons, for some atterns, we can t observe the full feature vector descrbed n the robablty structure. Only artal nformaton of the feature vector s observed, but some comonents are mssng. How to classfy such corruted nuts to obtan mnmum average error? Let the full feature vector [g,b], g reresents the observed or good features, b reresents the mssng or bad ones. In ths case, the otmal decson rule s constructed based on the osteror, g, as follows: arg max g

22 Mssng features/dataii Where b b b b g b b g b b g b b g b g g b b g g g g d d d, d, 2 d d d,, d,,,

23 Otmal Bayes decson rule s not achevable n ractce The otmal Bayes decson rule s not feasble n ractce. In any ractcal roblem, we can not have a comlete knowledge about the roblem. e.g., the class-condtonal robablty densty functons.d.f.,, are always unavalable and extremely hard to estmate. However, ossble to collect a set of samle data a set of feature observatons for each class n queston. The samle data are always far from enough to estmate a relable.d.f. by usng samle data themselves ONLY, e.g., some nonarametrc methods " samlng densty / hstogram. Queston: How to buld a reasonable classfer based on a lmted set of samle data, nstead of the true.d.f. s?

24 Statstcal Data Modelng For any real roblem, the true.d.f. s are always unknown, nether the functon form nor the arameters. Our aroach statstcal data modelng : based on the avalable samle data set, choose a roer statstcal model to ft nto the avalable data set. Data Modelng stage: once the statstcal model s selected, ts functon form becomes known excet a set of model arameters assocated wth the model are unknown to us. Learnng tranng stage: the unknown arameters can be estmated by fttng the model nto the data set based on certan estmaton crteron. the estmated statstcal model assumed model format + estmated arameters wll gve a arametrc.d.f. to aroxmate the real but unknown.d.f. of each class. Decson test stage: the estmated.d.f. s are lugged nto the otmal Bayes decson rule n lace of the real.d.f. s! lug-n MA decson rule Not otmal any more but erforms reasonably well n ractce Theoretcal analyss:! Statstcal Learnng Theory

25 Data modelng Estmated model.d.f. for class 2 λ2x Estmated model.d.f. for class λx

26 lug-n MA decson rule Once the statstcal models are estmated, they are treated as f they were true dstrbutons of the data, and lug nto the form of the otmal Bayes MA decson rule n lace of the unknown true.d.f. s. The lug-n MA decson rule: arg max arg max arg max arg max Λ Γ

27 Some useful modelsi A roer model must be chosen based on the nature of observaton data. Some useful statstcal models for a varety of data tyes: Normal Gaussan dstrbuton! un-modal contnuous feature scalars Multvarate normal Gaussan dstrbuton! un-modal contnuous feature vectors Gaussan Mxture models GMM! contnuous feature scalars/vectors wth mult-modal dstrbuton nature! For seaker recognton/verfcaton dstrbuton of seech features over a large oulaton

28 Some useful models II Some useful models cont d Markov chan model: dscrete sequental data N-gram model n language modelng Hdden Markov Models HMM: deal for varous knds of sequental observaton data; rovdes better modelng caablty than smle Markov chan model. Model seech sgnals for recognton one of the most successful story of data modelng Model language/text data for art-of-seech taggng, shallow language understandng, etc. Model bologcal data DNA & roten sequence: rofle HMM. Lots of other alcaton domans.

29 Some useful models III Some useful models cont d Random Markov Feld: mult-dmensonal satal data Model mage data: e.g., used for OCR, etc. HMM s a secal case of random Markov feld Grahcal models a.k.a., Bayesan networks, Belef networks Hgh-dmensonal data dscrete or contnuous To model a very comlex stochastc rocess Automatcally learn deendency from data Used wdely n machne learnng, data mnng HMM s also a secal case of grahcal model Neural networks, suort vector machne SVM DON T ft here. Not to model the dstrbuton.d.f. of data drectly. Dscrmnatve method: model the boundares of data sets

30 Generatve vs. dscrmnatve models osteror robablty lays the key role n attern classfcaton. Generatve Models: focus on robablty dstrbuton of data ~ the lug-n MA rule Dscrmnatve Models: drectly model dscrmnant functon: ~ g