Need for Probabilistic Reasoning. Raymond J. Mooney. Conditional Probability. Axioms of Probability Theory. Classification (Categorization)

Transcription

1 Need for Probablstc Reasonng CS 343: Artfcal Intelence Probablstc Reasonng and Naïve Bayes Rayond J. Mooney Unversty of Texas at Austn Most everyday reasonng s based on uncertan evdence and nferences. Classcal logc, whch only allows conclusons to be strctly true or strctly false, does not account for ths uncertanty or the need to wegh and cobne conflctng evdence. Straghtforward applcaton of probablty theory s practcal snce the large nuber of probablty paraeters requ are rarely, f ever, avalable. Therefore, early expert systes eployed farly ad hoc ethods for reasonng under uncertanty and for cobnng evdence. Recently, ethods ore rgorously founded n probablty theory that attept to decrease the aount of condtonal probabltes requ have flourshed. 2 Axos of Probablty Theory All probabltes between 0 and 0 True proposton has probablty, false has probablty 0. true) false) 0. The probablty of dsjuncton s: A + A A A B B Condtonal Probablty A s the probablty of A gven B Assues that B s all and only nforaton nown. Defned by: A A A A B B 3 4 Independence A and B are ndependent ff: P ( A These two constrants are logcally equvalent P ( B Therefore, f A and B are ndependent: A P ( A P ( A Classfcaton (Categorzaton) Gven: A descrpton of an nstance, x X, where X s the nstance language or nstance space. A fxed set of categores: C{c, c 2, c n } Deterne: The category of x: c(x) C, where c(x) s a categorzaton functon whose doan s X and whose range s C. If c(x) s a bnary functon C{0,} ({true,false}, {postve, negatve}) then t s called a concept. 5 6

2 Learnng for Categorzaton Saple Learnng Proble A tranng exaple s an nstance x X, pa wth ts correct category c(x): <x, c(x)> for an unnown categorzaton functon, c. Gven a set of tranng exaples, D. Fnd a hypotheszed categorzaton functon, h(x), such that: < x, c( x) > D : h( x) c( x) Consstency 7 Instance language: <sze, color, shape> sze {sall, edu, large} color {,, green} shape {square, crcle, trangle} C {postve, negatve} D: Exaple Sze Color Shape sall crcle postve 2 large crcle postve 3 sall trangle negatve 4 large crcle negatve 8 Jont Dstrbuton Probablstc Classfcaton The jont probablty dstrbuton for a set of rando varables, X,,X n gves the probablty of every cobnaton of values (an n- densonal array wth v n values f all varables are dscrete wth v values, all v n values ust su to ): X,,X n ) postve negatve crcle square crcle square The probablty of all possble conjunctons (assgnents of values to soe subset of varables) can be calculated by sung the approprate subset of values fro the jont dstrbuton. crcle) ) Therefore, all condtonal probabltes can also be calculated. postve crcle) 0.20 postve crcle) 0.80 crcle) Let Y be the rando varable for the class whch taes values {y,y 2, y }. Let X be the rando varable descrbng an nstance consstng of a vector of values for n features <X,X 2 X n >, let x be a possble value for X and x j a possble value for X. For classfcaton, we need to copute Yy Xx ) for However, gven no other assuptons, ths requres a table gvng the probablty of each category for each possble nstance n the nstance space, whch s possble to accurately estate fro a reasonably-szed tranng set. Assung Y and all X are bnary, we need 2 n entres to specfy Ypos Xx ) for each of the 2 n possble x s snce Yneg Xx ) Ypos Xx ) Copa to 2 n+ entres for the jont dstrbuton Y,X,X 2 X n ) 0 Bayes Theore Bayesan Categorzaton E P ( H E) E) Sple proof fro defnton of condtonal probablty: H E) H E) (Def. cond. prob.) E) H E) E (Def. cond. prob.) P ( H E) E E QED: P ( H E) E) Deterne category of x by deternng for each y Y y ) X x Y y) Y y X x ) X x ) Xx ) can be deterned snce categores are coplete and dsjont. Y y X x ) X x ) Y y ) X x Y y ) X x ) Y y ) X x Y y ) 2 2

3 Bayesan Categorzaton (cont.) Need to now: Prors: Yy ) Condtonals: Xx Yy ) Yy ) are easly estated fro data. If n of the exaples n D are n y then Yy ) n / D Too any possble nstances (e.g. 2 n for bnary features) to estate all Xx Yy ). Stll need to ae soe sort of ndependence assuptons about the features to ae learnng tractable. 3 Generatve Probablstc Models Assue a sple (usually unrealstc) probablstc ethod by whch the data was generated. For categorzaton, each category has a dfferent paraeterzed generatve odel that characterzes that category. Tranng: Use the data for each category to estate the paraeters of the generatve odel for that category. Maxu Lelhood Estaton (MLE): Set paraeters to axze the probablty that the odel produced the gven tranng data. If M λ denotes a odel wth paraeter values λ and D s the tranng data for the th class, fnd odel paraeters for class (λ ) that axze the lelhood of D : λ argax D M ) λ Testng: Use Bayesan analyss to deterne the category odel that ost lely generated a specfc test nstance. λ 4 Naïve Bayes Generatve Model Naïve Bayes Inference Proble neg pos pos crc???? neg pos pos ed s grn crc tr tr crc crc crc crc ed s s crc tr crc crctr tr Sze Color Shape Sze Color Shape Postve Negatve grn grn grn 5 ed s grn crc tr tr crc crc crc crc ed s s crc tr crc crctr tr Sze Color Shape Sze Color Shape Postve Negatve grn grn grn 6 Naïve Bayesan Categorzaton Naïve Bayes Categrzaton Exaple If we assue features of an nstance are ndependent gven the category (condtonally ndependent). X Y ) X, X 2, L X n Y ) X Y ) Therefore, we then only need to now X Y) for each possble par of a feature-value and a category. If Y and all X and bnary, ths requres specfyng only 2n paraeters: X true Ytrue) and X true Yfalse) for each X X false Y) X true Y) Copa to specfyng 2 n paraeters wthout any ndependence assuptons. Probablty postve negatve Y) sall Y) edu Y) large Y) Y) Y) green Y) square Y) trangle Y) crcle Y) Test Instance: <edu,, crcle> 7 8 3

4 Naïve Bayes Categorzaton Exaple Naïve Bayes Dagnoss Exaple Probablty postve negatve Y) edu Y) Y) crcle Y) postve X) postve)*edu postve)* postve)*crcle postve) / X) 0.5 * 0. * 0.9 * / X) / negatve X) negatve)*edu negatve)* negatve)*crcle negatve) / X) 0.5 * 0.2 * 0.3 * / X) / postve X) + negatve X) / X) / X) X) ( ) Test Instance: <edu,, crcle> 9 C {allergy, cold, well} e sneeze; e 2 cough; e 3 fever E {sneeze, cough, fever} Prob Well Cold Allergy c ) sneeze c ) cough c ) fever c ) Naïve Bayes Dagnoss Exaple (cont.) Estatng Probabltes Probablty Well Cold Allergy c ) sneeze c ) cough c ) fever c ) well E) (0.9)(0.)(0.)(0.99)/E)0.0089/E) cold E) (0.05)(0.9)(0.8)(0.3)/E)0.0/E) allergy E) (0.05)(0.9)(0.7)(0.6)/E)0.09/E) Most probable category: allergy E) well E) 0.23 cold E) 0.26 allergy E) 0.50 E{sneeze, cough, fever} 2 Norally, probabltes are estated based on observed frequences n the tranng data. If D contans n exaples n category y, and n j of these n exaples have the jth value for feature X, x j, then: nj P ( X xj Y y ) n However, estatng such probabltes fro sall tranng sets s error-prone. If due only to chance, a rare feature, X, s always false n the tranng data, y :X true Yy ) 0. If X true then occurs n a test exaple, X, the result s that y : X Yy ) 0 and y : Yy X) 0 22 Probablty Estaton Exaple Ex Sze Color Shape sall crcle postve 2 large crcle postve 3 sall trangle negtve 4 large crcle negtve Test Instance X: <edu,, crcle> postve X) 0.5 * 0.0 *.0 *.0 / X) 0 Probablty postve negatve Y) sall Y) edu Y) large Y) Y) Y) green Y) square Y) trangle Y) crcle Y) Soothng To account for estaton fro sall saples, probablty estates are adjusted or soothed. Laplace soothng usng an -estate assues that each feature s gven a pror probablty, p, that s assued to have been prevously observed n a vrtual saple of sze. nj + p X xj Y y ) n + For bnary features, p s sply assued to be 0.5. negatve X) 0.5 * 0.0 * 0.5 * 0.5 / X)

5 Laplace Sothng Exaple Text Categorzaton Applcatons Assue tranng set contans 0 postve exaples: 4: sall 0: edu 6: large Estate paraeters as follows (f, p/3) sall postve) (4 + /3) / (0 + ) edu postve) (0 + /3) / (0 + ) 0.03 large postve) (6 + /3) / (0 + ) sall or edu or large postve).0 25 Web pages Recoendng Yahoo-le classfcaton Newsgroup/Blog Messages Recoendng flterng Sentent analyss for aretng News artcles Personalzed newspaper Eal essages Routng Prortzng Folderzng flterng Advertsng on Gal 26 Text Categorzaton Methods Naïve Bayes for Text Most coon representaton of a docuent s a bag of words,.e. set of words wth ther frequences, word order s gno. Gves a hgh-densonal vector representaton (one feature for each word). Vectors are sparse snce ost words are rare. Zpf s law and heavy-taled dstrbutons Modeled as generatng a bag of words for a docuent n a gven category by repeatedly saplng wth replaceent fro a vocabulary V {w, w 2, w } based on the probabltes w j c ). Sooth probablty estates wth Laplace -estates assung a unfor dstrbuton over all words (p / V ) and V Equvalent to a vrtual saple of seeng each word n each category exactly once Naïve Bayes Generatve Model for Text Naïve Bayes Text Classfcaton Vagra wn hot Ngera deal lottery nude $ Vagra legt legt legt legt scence PM coputer Frday test hoewor March score May exa legt 29 Vagra wn hot Ngera deal lottery nude $ Vagra Wn lotttery $???? legt legt legt legt scence PM coputer Frday test hoewor March score May exa legt 30 5

6 Text Naïve Bayes Aorth (Tran) Let V be the vocabulary of all words n the docuents n D For each category c C Let D be the subset of docuents n D n category c c ) D / D Let T be the concatenaton of all the docuents n D Let n be the total nuber of word occurrences n T For each word w j V Let n j be the nuber of occurrences of w j n T Let w j c ) (n j + ) / (n + V ) Text Naïve Bayes Aorth (Test) Gven a test docuent X Let n be the nuber of word occurrences n X Return the category: argax c ) c C n a c ) where a s the word occurrng the th poston n X 3 32 Underflow Preventon Multplyng lots of probabltes, whch are between 0 and by defnton, can result n floatng-pont underflow. Snce log(xy) log(x) + log(y), t s better to perfor all coputatons by sung logs of probabltes rather than ultplyng probabltes. Class wth hghest fnal un-noralzed log probablty score s stll the ost probable. Coents on Naïve Bayes Maes probablstc nference tractable by ang a strong assupton of condtonal ndependence. Tends to wor farly well despte ths strong assupton. Experents show t to be qute copettve wth other classfcaton ethods on standard datasets. Partcularly popular for text categorzaton, e.g. flterng