Bayesian Learning. Bayes Theorem. MAP, MLhypotheses. MAP learners. Minimum description length principle. Bayes optimal classier. Naive Bayes learner

Transcription

1 Bayesian Learning [Read Ch. 6] [Suggested exercises: 6.1, 6.2, 6.6] Bayes Theorem MAP, MLhypotheses MAP learners Minimum description length principle Bayes optimal classier Naive Bayes learner Example: Learning over text data Bayesian belief networks Expectation Maximization algorithm 125 lecture slides for textbook Machine Learning, T. Mitchell, McGraw Hill, 1997

2 Two Roles for Bayesian Methods Provides practical learning algorithms: Naive Bayes learning Bayesian belief network learning Combine prior knowledge (prior probabilities) with observed data Requires prior probabilities Provides useful conceptual framework Provides \gold standard" for evaluating other learning algorithms Additional insight into Occam's razor 126 lecture slides for textbook Machine Learning, T. Mitchell, McGraw Hill, 1997

3 Bayes Theorem P (hjd) = P (Djh)P (h) P (D) P (h) = prior probability of hypothesis h P (D) = prior probability of training data D P (hjd) = probability ofh given D P (Djh) = probability ofd given h 127 lecture slides for textbook Machine Learning, T. Mitchell, McGraw Hill, 1997

4 Choosing Hypotheses P (hjd) = P (Djh)P (h) P (D) Generally want the most probable hypothesis given the training data Maximum a posteriori hypothesis h MAP : h MAP = arg max h2h P (hjd) = arg max h2h = arg max h2h P (Djh)P (h) P (D) P (Djh)P (h) If assume P (h i )=P (h j ) then can further simplify, and choose the Maximum likelihood (ML) hypothesis h ML = arg max h i 2H P (Djh i) 128 lecture slides for textbook Machine Learning, T. Mitchell, McGraw Hill, 1997

5 Bayes Theorem Does patient have cancer or not? A patient takes a lab test and the result comes back positive. The test returns a correct positive result in only 98% of the cases in which the disease is actually present, and a correct negative result in only 97% of the cases in which the disease is not present. Furthermore, :008 of the entire population have this cancer. P (cancer) = P (:cancer) = P (+jcancer) = P (;jcancer) = P (+j:cancer) = P (;j:cancer) = 129 lecture slides for textbook Machine Learning, T. Mitchell, McGraw Hill, 1997

6 Basic Formulas for Probabilities Product Rule: probability P (A ^ B) of a conjunction of two events A and B: P (A ^ B) =P (AjB)P (B) =P (BjA)P (A) Sum Rule: probability of a disjunction of two events A and B: P (A _ B) =P (A)+P (B) ; P (A ^ B) Theorem of total probability: if events A 1 ::: A n are mutually exclusive with P n i=1 P (A i) = 1, then P (B) = n X i=1 P (BjA i )P (A i ) 130 lecture slides for textbook Machine Learning, T. Mitchell, McGraw Hill, 1997

7 Brute Force MAP Hypothesis Learner 1. For each hypothesis h in H, calculate the posterior probability P (hjd) = P (Djh)P (h) P (D) 2. Output the hypothesis h MAP with the highest posterior probability h MAP = argmax h2h P (hjd) 131 lecture slides for textbook Machine Learning, T. Mitchell, McGraw Hill, 1997

8 Relation to Concept Learning Consider our usual concept learning task instance space X, hypothesis space H, training examples D consider the FindS learning algorithm (outputs most specic hypothesis from the version space VS H D ) What would Bayes rule produce as the MAP hypothesis? Does FindS output a MAP hypothesis?? 132 lecture slides for textbook Machine Learning, T. Mitchell, McGraw Hill, 1997

9 Relation to Concept Learning Assume xed set of instances hx 1 ::: x m i Assume D is the set of classications D = hc(x 1 ) ::: c(x m )i Choose P (Djh): 133 lecture slides for textbook Machine Learning, T. Mitchell, McGraw Hill, 1997

10 Relation to Concept Learning Assume xed set of instances hx 1 ::: x m i Assume D is the set of classications D = hc(x 1 ) ::: c(x m )i Choose P (Djh) P (Djh) = 1 if h consistent with D P (Djh) = 0 otherwise Choose P (h) to be uniform distribution P (h) = 1 jhj Then, P (hjd) = for all h in H 8 >< >: 1 jv S H D j if h is consistent with D 0 otherwise 134 lecture slides for textbook Machine Learning, T. Mitchell, McGraw Hill, 1997

11 Evolution of Posterior Probabilities P( h) P(h D1) P(h D1, D2) hypotheses hypotheses hypotheses ( a) ( b) ( c) 135 lecture slides for textbook Machine Learning, T. Mitchell, McGraw Hill, 1997

12 Characterizing Learning Algorithms by Equivalent MAP Learners Inductive system Training examples D Hypothesis space H Candidate Elimination Algorithm Output hypotheses Equivalent Bayesian inference system Training examples D Hypothesis space H P(h) uniform P(D h) = 0 if inconsistent, = 1 if consistent Brute force MAP learner Output hypotheses Prior assumptions made explicit 136 lecture slides for textbook Machine Learning, T. Mitchell, McGraw Hill, 1997

13 Learning A Real Valued Function y f e h ML Consider any real-valued target function f Training examples hx i d i i,whered i is noisy training value d i = f(x i )+e i e i is random variable (noise) drawn independently for each x i according to some Gaussian distribution with mean=0 Then the maximum likelihood hypothesis h ML is the one that minimizes the sum of squared errors: h ML = arg min h2h mx i=1 x (d i ; h(x i )) lecture slides for textbook Machine Learning, T. Mitchell, McGraw Hill, 1997

14 Learning A Real Valued Function h ML = argmax h2h = argmax h2h = argmax h2h p(djh) my i=1 my i=1 p(d i jh) 1 p e; ( d i ;h(x i ) 2 Maximize natural log of this instead... ) 2 h ML = argmax h2h = argmax h2h = argmax h2h = argmin h2h mx 1 ln p ; 1 i= mx ; 1 0 d B i ; h(x i i=1 2 mx i=1 mx i=1 0 ; (d i ; h(x i )) 2 (d i ; h(x i )) 2 1 C A d i ; h(x i ) 2 1 C A lecture slides for textbook Machine Learning, T. Mitchell, McGraw Hill, 1997

15 Learning to Predict Probabilities Consider predicting survival probability from patient data Training examples hx i d i i,whered i is 1 or 0 Want to train neural network to output a probability given x i (not a 0 or 1) In this case can show h ML = argmax h2h mx i=1 d i ln h(x i )+(1; d i )ln(1; h(x i )) Weight update rule for a sigmoid unit: where w jk w jk = m X i=1 w jk +w jk (d i ; h(x i )) x ijk 139 lecture slides for textbook Machine Learning, T. Mitchell, McGraw Hill, 1997

16 Minimum Description Length Principle Occam's razor: prefer the shortest hypothesis MDL: prefer the hypothesis h that minimizes h MDL = argmin L C1 (h)+l C2 (Djh) h2h where L C (x) is the description length of x under encoding C Example: H = decision trees, D = training data labels L C1 (h) is # bits to describe tree h L C2 (Djh) is # bits to describe D given h { Note L C2 (Djh) = 0 if examples classied perfectly by h. Need only describe exceptions Hence h MDL trades o tree size for training errors 140 lecture slides for textbook Machine Learning, T. Mitchell, McGraw Hill, 1997

17 Minimum Description Length Principle h MAP = arg max h2h P (Djh)P (h) = arg max h2h log 2 P (Djh) + log 2 P (h) = arg min h2h ; log 2 P (Djh) ; log 2 Interesting fact from information theory: P (h) (1) The optimal (shortest expected coding length) code for an event with probability p is ; log 2 p bits. So interpret (1): ;log 2 P (h) is length of h under optimal code ;log 2 P (Djh) is length of D given h under optimal code! prefer the hypothesis that minimizes length(h)+length(misclassif ications) 141 lecture slides for textbook Machine Learning, T. Mitchell, McGraw Hill, 1997

18 Most Probable Classication of New Instances So far we've sought the most probable hypothesis given the data D (i.e., h MAP ) Given new instance x, what is its most probable classication? h MAP (x) is not the most probable classication! Consider: Three possible hypotheses: P (h 1 jd) =:4 P (h 2 jd) =:3 P (h 3 jd) =:3 Given new instance x, h 1 (x) =+ h 2 (x) =; h 3 (x) =; What's most probable classication of x? 142 lecture slides for textbook Machine Learning, T. Mitchell, McGraw Hill, 1997

19 Bayes Optimal Classier Bayes optimal classication: Example: arg max v j 2V X h i 2H P (v j jh i )P (h i jd) therefore and P (h 1 jd) =:4 P (;jh 1 )=0 P (+jh 1 )=1 P (h 2 jd) =:3 P (;jh 2 )=1 P (+jh 2 )=0 P (h 3 jd) =:3 P (;jh 3 )=1 P (+jh 3 )=0 X h i 2H X h i 2H arg max v j 2V P (+jh i )P (h i jd) = :4 P (;jh i )P (h i jd) = :6 X h i 2H P (v j jh i )P (h i jd) = ; 143 lecture slides for textbook Machine Learning, T. Mitchell, McGraw Hill, 1997

20 Gibbs Classier Bayes optimal classier provides best result, but can be expensive if many hypotheses. Gibbs algorithm: 1. Choose one hypothesis at random, according to P (hjd) 2. Use this to classify new instance Surprising fact: Assume target concepts are drawn at random from H according to priors on H. Then: E[error Gibbs ] 2E[error BayesOptimal ] Suppose correct, uniform prior distribution over H, then Pick any hypothesis from VS, with uniform probability Its expected error no worse than twice Bayes optimal 144 lecture slides for textbook Machine Learning, T. Mitchell, McGraw Hill, 1997

21 Naive Bayes Classier Along with decision trees, neural networks, nearest nbr, one of the most practical learning methods. When to use Moderate or large training set available Attributes that describe instances are conditionally independent given classication Successful applications: Diagnosis Classifying text documents 145 lecture slides for textbook Machine Learning, T. Mitchell, McGraw Hill, 1997

22 Naive Bayes Classier Assume target function f : X! V, where each instance x described by attributes ha 1 a 2 :::a n i. Most probable value of f(x) is: v MAP v MAP = argmax v j 2V = argmax v j 2V = argmax v j 2V P (v j ja 1 a 2 :::a n ) P (a 1 a 2 :::a n jv j )P (v j ) P (a 1 a 2 :::a n ) P (a 1 a 2 :::a n jv j )P (v j ) Naive Bayes assumption: P (a 1 a 2 :::a n jv j )= Y i P (a i jv j ) which gives Naive Bayes classier: v NB = argmax v j 2V P (v j ) Y i P (a i jv j ) 146 lecture slides for textbook Machine Learning, T. Mitchell, McGraw Hill, 1997

23 Naive Bayes Algorithm Naive Bayes Learn(examples) For each targetvalue v j ^P (v j ) estimate P (v j ) For each attribute value a i of each attribute a ^P (a i jv j ) estimate P (a i jv j ) Classify New Instance(x) v NB = argmax v j 2V ^P (v j ) Y a i 2x ^P (a i jv j ) 147 lecture slides for textbook Machine Learning, T. Mitchell, McGraw Hill, 1997

24 Naive Bayes: Example Consider PlayTennis again, and new instance houtlk = sun T emp = cool Humid = high W ind = strong Want to compute: v NB = argmax v j 2V P (v j ) Y i P (a i jv j ) P (y) P (sunjy) P (cooljy) P (highjy) P (strongjy) =:005 P (n) P (sunjn) P (cooljn) P (highjn) P (strongjn) =:021! v NB = n 148 lecture slides for textbook Machine Learning, T. Mitchell, McGraw Hill, 1997

25 Naive Bayes: Subtleties 1. Conditional independence assumption is often violated P (a 1 a 2 :::a n jv j )= Y i P (a i jv j )...but it works surprisingly well anyway. Note don't need estimated posteriors ^P (vj jx) tobe correct need only that argmax v j 2V ^P (v j ) Y i ^P (a i jv j ) = argmax v j 2V P (v j )P (a 1 ::: a n jv j ) see [Domingos &Pazzani, 1996] for analysis Naive Bayes posteriors often unrealistically close to 1 or lecture slides for textbook Machine Learning, T. Mitchell, McGraw Hill, 1997

26 Naive Bayes: Subtleties 2. what if none of the training instances with target value v j have attribute value a i? Then ^P (a i jv j ) = 0, and... ^P (v j ) Y i ^P (a i jv j )=0 Typical solution is Bayesian estimate for ^P (ai jv j ) where ^P (a i jv j ) n c + mp n + m n is number of training examples for which v = v j, n c number of examples for which v = v j and a = a i p is prior estimate for ^P (a i jv j ) m is weight given to prior (i.e. number of \virtual" examples) 150 lecture slides for textbook Machine Learning, T. Mitchell, McGraw Hill, 1997

27 Learning to Classify Text Why? Learn which news articles are of interest Learn to classify web pages by topic Naive Bayes is among most eective algorithms What attributes shall we use to represent text documents?? 151 lecture slides for textbook Machine Learning, T. Mitchell, McGraw Hill, 1997

28 Learning to Classify Text Target concept Interesting? :Document!f+ ;g 1. Represent each document by vector of words one attribute per word position in document 2. Learning: Use training examples to estimate P (+) P (;) P (docj+) P (docj;) Naive Bayes conditional independence assumption P (docjv j )= length(doc) Y i=1 P (a i = w k jv j ) where P (a i = w k jv j ) is probability that word in position i is w k, given v j one more assumption: P (a i = w k jv j )=P (a m = w k jv j ) 8i m 152 lecture slides for textbook Machine Learning, T. Mitchell, McGraw Hill, 1997

29 Learn naive Bayes text(examples V ) 1. collect all words and other tokens that occur in Examples all distinct words and other tokens in Examples V ocabulary 2. calculate the required P (v j ) and P (w k jv j ) probability terms For each targetvalue v j in V do { docs j subset of Examples for which the target value is v j { P (v j ) jdocs j j jexamplesj { Text j a single document created by concatenating all members of docs j { n total number of words in Text j (counting duplicate words multiple times) { for each word w k in V ocabulary number of times word w k occurs in n k Text j P (w k jv j ) n k +1 n+jv ocabularyj 153 lecture slides for textbook Machine Learning, T. Mitchell, McGraw Hill, 1997

30 Classify naive Bayes text(doc) all word positions in Doc that contain tokens found in V ocabulary positions Return v NB,where v NB = argmax v j 2V P (v j ) Y i2positions P (a i jv j ) 154 lecture slides for textbook Machine Learning, T. Mitchell, McGraw Hill, 1997

31 Twenty NewsGroups Given 1000 training documents from each group Learn to classify new documents according to which newsgroup it came from comp.graphics comp.os.ms-windows.misc comp.sys.ibm.pc.hardware comp.sys.mac.hardware comp.windows.x alt.atheism soc.religion.christian talk.religion.misc talk.politics.mideast talk.politics.misc talk.politics.guns misc.forsale rec.autos rec.motorcycles rec.sport.baseball rec.sport.hockey sci.space sci.crypt sci.electronics sci.med Naive Bayes: 89% classication accuracy 155 lecture slides for textbook Machine Learning, T. Mitchell, McGraw Hill, 1997

32 Article from rec.sport.hockey Path: cantaloupe.srv.cs.cmu.edu!das-news.harvard.e From: (John Doe) Subject: Re: This year's biggest and worst (opinio Date: 5 Apr 93 09:53:39 GMT I can only comment on the Kings, but the most obvious candidate for pleasant surprise is Alex Zhitnik. He came highly touted as a defensive defenseman, but he's clearly much more than that. Great skater and hard shot (though wish he were more accurate). In fact, he pretty much allowed the Kings to trade away that huge defensive liability Paul Coffey. Kelly Hrudey is only the biggest disappointment if you thought he was any good to begin with. But, at best, he's only a mediocre goaltender. A better choice would be Tomas Sandstrom, though not through any fault of his own, but because some thugs in Toronto decided 156 lecture slides for textbook Machine Learning, T. Mitchell, McGraw Hill, 1997

33 Learning Curve for 20 Newsgroups News Bayes TFIDF PRTFIDF Accuracy vs. Training set size (1/3 withheld for test) 157 lecture slides for textbook Machine Learning, T. Mitchell, McGraw Hill, 1997

34 Bayesian Belief Networks Interesting because: Naive Bayes assumption of conditional independence too restrictive But it's intractable without some such assumptions... Bayesian Belief networks describe conditional independence among subsets of variables! allows combining prior knowledge about (in)dependencies among variables with observed training data (also called Bayes Nets) 158 lecture slides for textbook Machine Learning, T. Mitchell, McGraw Hill, 1997

35 Conditional Independence Denition: X is conditionally independent of Y given Z if the probability distribution governing X is independent of the value of Y given the value of Z thatis,if (8x i y j z k ) P (X = x i jy = y j Z = z k )=P (X = x i jz = z k more compactly, we write P (XjY Z)=P (XjZ) Example: T hunder is conditionally independent of Rain, given Lightning P (T hunderjrain Lightning)=P (T hunderjlightning) Naive Bayes uses cond. indep. to justify P (X Y jz) = P (XjY Z)P (Y jz) = P (XjZ)P (Y jz) 159 lecture slides for textbook Machine Learning, T. Mitchell, McGraw Hill, 1997

36 Bayesian Belief Network Storm BusTourGroup S,B S, B S,B S, B Lightning Campfire C C Campfire Thunder ForestFire Network represents a set of conditional independence assertions: Each node is asserted to be conditionally independent of its nondescendants, given its immediate predecessors. Directed acyclic graph 160 lecture slides for textbook Machine Learning, T. Mitchell, McGraw Hill, 1997

37 Bayesian Belief Network Storm BusTourGroup S,B S, B S,B S, B Lightning Campfire C C Campfire Thunder ForestFire Represents joint probability distribution over all variables e.g., P (Storm BusT ourgroup : : : F orestf ire) in general, P (y 1 ::: y n )= n Y i=1 P (y i jparents(y i )) where P arents(y i ) denotes immediate predecessors of Y i in graph so, joint distribution is fully dened by graph, plus the P (y i jparents(y i )) 161 lecture slides for textbook Machine Learning, T. Mitchell, McGraw Hill, 1997

38 Inference in Bayesian Networks How can one infer the (probabilities of) values of one or more network variables, given observed values of others? Bayes net contains all information needed for this inference If only one variable with unknown value, easy to infer it In general case, problem is NP hard In practice, can succeed in many cases Exact inference methods work well for some network structures Monte Carlo methods \simulate" the network randomly to calculate approximate solutions 162 lecture slides for textbook Machine Learning, T. Mitchell, McGraw Hill, 1997

39 Learning of Bayesian Networks Several variants of this learning task Network structure might be known or unknown Training examples might provide values of all network variables, or just some If structure known and observe all variables Then it's easy as training a Naive Bayes classier 163 lecture slides for textbook Machine Learning, T. Mitchell, McGraw Hill, 1997

40 Learning Bayes Nets Suppose structure known, variables partially observable e.g., observe ForestFire, Storm, BusTourGroup, Thunder, but not Lightning, Campre... Similar to training neural network with hidden units In fact, can learn network conditional probability tables using gradient ascent! Converge to network h that (locally) maximizes P (Djh) 164 lecture slides for textbook Machine Learning, T. Mitchell, McGraw Hill, 1997

41 Gradient Ascent for Bayes Nets Let w ijk denote one entry in the conditional probability table for variable Y i in the network w ijk = P (Y i = y ij jp arents(y i ) = the list u ik of values) e.g., if Y i = Campfire,thenu ik might be hstorm = T BusT ourgroup = F i Perform gradient ascent by repeatedly 1. update all w ijk using training data D w ijk w ijk + X d2d P h (y ij u ik jd) w ijk 2. then, renormalize the w ijk to assure P j w ijk =1 0 w ijk lecture slides for textbook Machine Learning, T. Mitchell, McGraw Hill, 1997

42 More on Learning Bayes Nets EM algorithm can also be used. Repeatedly: 1. Calculate probabilities of unobserved variables, assuming h 2. Calculate new w ijk to maximize E[ln P (Djh)] where D now includes both observed and (calculated probabilities of) unobserved variables When structure unknown... Algorithms use greedy search to add/substract edges and nodes Active research topic 166 lecture slides for textbook Machine Learning, T. Mitchell, McGraw Hill, 1997

43 Summary: Bayesian Belief Networks Combine prior knowledge with observed data Impact of prior knowledge (when correct!) is to lower the sample complexity Active research area { Extend from boolean to real-valued variables { Parameterized distributions instead of tables { Extend to rst-order instead of propositional systems { More eective inference methods { lecture slides for textbook Machine Learning, T. Mitchell, McGraw Hill, 1997

44 Expectation Maximization (EM) When to use: Data is only partially observable Unsupervised clustering (target value unobservable) Supervised learning (some instance attributes unobservable) Some uses: Train Bayesian Belief Networks Unsupervised clustering (AUTOCLASS) Learning Hidden Markov Models 168 lecture slides for textbook Machine Learning, T. Mitchell, McGraw Hill, 1997

45 Generating Data from Mixture of k Gaussians p(x) Each instance x generated by 1. Choosing one of the k Gaussians with uniform probability 2. Generating an instance at random according to that Gaussian x 169 lecture slides for textbook Machine Learning, T. Mitchell, McGraw Hill, 1997

46 EM for Estimating k Means Given: Instances from X generated by mixture of k Gaussian distributions Unknown means h 1 ::: k i of the k Gaussians Don't know which instance x i was generated by which Gaussian Determine: Maximum likelihood estimates of h 1 ::: k i Think of full description of each instance as y i = hx i z i1 z i2 i, where z ij is 1 if x i generated by jth Gaussian x i observable z ij unobservable 170 lecture slides for textbook Machine Learning, T. Mitchell, McGraw Hill, 1997

47 EM for Estimating k Means EM Algorithm: Pick random initial h = h 1 2 i, then iterate E step: Calculate the expected value E[z ij ]ofeach hidden variable z ij, assuming the current hypothesis h = h 1 2 i holds. p(x = x i j = j ) E[z ij ] = P 2n=1 p(x = x i j = n ) = e ; (x i ; j )2 P 2n=1 e ; (x i ; n) 2 M step: Calculate a new maximum likelihood hypothesis h 0 = h i, assuming the value taken on by each hidden variable z ij is its expected value E[z ij ] calculated above. Replace h = h 1 2 i by h 0 = h 0 1 0i. 2 j P m i=1 E[z ij] x i P mi=1 E[z ij ] 171 lecture slides for textbook Machine Learning, T. Mitchell, McGraw Hill, 1997

48 EM Algorithm Converges to local maximum likelihood h and provides estimates of hidden variables z ij In fact, local maximum in E[ln P (Y jh)] Y is complete (observable plus unobservable variables) data Expected value is taken over possible values of unobserved variables in Y 172 lecture slides for textbook Machine Learning, T. Mitchell, McGraw Hill, 1997

49 General EM Problem Given: Observed data X = fx 1 ::: x m g Unobserved data Z = fz 1 ::: z m g Parameterized probability distribution P (Y jh), where { Y = fy 1 ::: y m g is the full data y i = x i [ z i { h are the parameters Determine: h that (locally) maximizes E[ln P (Y jh)] Many uses: Train Bayesian belief networks Unsupervised clustering (e.g., k means) Hidden Markov Models 173 lecture slides for textbook Machine Learning, T. Mitchell, McGraw Hill, 1997

50 General EM Method Dene likelihood function Q(h 0 jh) which calculates Y = X [ Z using observed X and current parameters h to estimate Z EM Algorithm: Q(h 0 jh) E[ln P (Y jh 0 )jh X] Estimation (E) step: Calculate Q(h 0 jh) using the current hypothesis h and the observed data X to estimate the probability distribution over Y. Q(h 0 jh) E[ln P (Y jh 0 )jh X] Maximization (M) step: Replace hypothesis h by the hypothesis h 0 that maximizes this Q function. h argmax Q(h 0 jh) h lecture slides for textbook Machine Learning, T. Mitchell, McGraw Hill, 1997