Lecture 8: Maximum Likelihood Estimation (MLE) (cont d.) Maximum a posteriori (MAP) estimation Naïve Bayes Classifier

Transcription

1 Lecture 8: Maximum Likelihood Estimation (MLE) (cont d.) Maximum a posteriori (MAP) estimation Naïve Bayes Classifier Aykut Erdem March 2016 Hacettepe University

2 Flipping a Coin Last time Flipping a Coin I have a coin, if I flip it, what s the probability that it will fall with the head up? Let us flip it a few times to estimate the probability: slide by Barnabás Póczos & Alex Smola The estimated probability is: 3/5 Frequency of heads 13 2

3 Last time Flipping a Coin The estimated probability is: 3/5 Frequency of heads slide by Barnabás Póczos & Alex Smola Questions: (1) Why frequency of heads??? (2) How good is this estimation??? (3) Why is this a machine learning problem??? We are going to answer these questions 3

4 Question (1) Why frequency of heads??? Frequency of heads is exactly the maximum likelihood estimator for this problem MLE has nice properties (interpretation, statistical guarantees, simple) slide by Barnabás Póczos & Alex Smola 4

5 MLE distribution MLEfor forbernoulli Bernoulli distribution Data, D = P(Heads) = θ, P(Tails) = 1-θ Flips are i.i.d.: slide by Barnabás Póczos & Alex Smola Independent events Identically distributed according to Bernoulli distribution MLE: Choose θ that maximizes the probability of observed data 5 17

6 MLE distribution MLEfor forbernoulli Bernoulli distribution Data, D = P(Heads) = θ, P(Tails) = 1-θ Flips are i.i.d.: slide by Barnabás Póczos & Alex Smola Independent events Identically distributed according to Bernoulli distribution MLE: Choose θ that maximizes the probability of observed data 6 17

7 MLE distribution MLEfor forbernoulli Bernoulli distribution Data, D = P(Heads) = θ, P(Tails) = 1-θ Flips are i.i.d.: slide by Barnabás Póczos & Alex Smola Independent events Identically distributed according to Bernoulli distribution MLE: Choose θ that maximizes the probability of observed data 7 17

8 MLE distribution MLEfor forbernoulli Bernoulli distribution Data, D = P(Heads) = θ, P(Tails) = 1-θ Flips are i.i.d.: slide by Barnabás Póczos & Alex Smola Independent events Identically distributed according to Bernoulli distribution MLE: Choose θ that maximizes the probability of observed data 8 17

9 Maximum Likelihood Maximum Likelihood Estimation Estimation MLE: Choose θ that maximizes the probability of observed data Maximum Likelihood Estimation Independent independentdraws draws ose θ that maximizes the probability of observed data iden,cally Identically distributed distributed slide by Barnabás Póczos & Alex Smola 18 9

10 Maximum Likelihood Maximum Likelihood Estimation Estimation MLE: Choose θ that maximizes the probability of observed data Maximum Likelihood Estimation Independent draws independent draws ose θ that maximizes the probability of observed data iden,cally Identically distributed distributed slide by Barnabás Póczos & Alex Smola 18 10

11 Maximum Likelihood Maximum Likelihood Estimation Estimation MLE: Choose θ that maximizes the probability of observed data Maximum Likelihood Estimation Independent draws independent draws ose θ that maximizes the probability of observed data identically Identically distributed distributed slide by Barnabás Póczos & Alex Smola 18 11

12 Maximum Likelihood Maximum Likelihood Estimation Estimation MLE: Choose θ that maximizes the probability of observed data Maximum Likelihood Estimation Independent draws independent draws ose θ that maximizes the probability of observed data identically Identically distributed distributed slide by Barnabás Póczos & Alex Smola 18 12

13 Maximum Likelihood Maximum Likelihood Estimation Estimation MLE: Choose θ that maximizes the probability of observed data Maximum Likelihood Estimation Independent draws independent draws ose θ that maximizes the probability of observed data identically Identically distributed distributed slide by Barnabás Póczos & Alex Smola 18 13

14 Maximum Maximum Likelihood Likelihood Estimation Maximum Estimation Likelihood Estimation Estimation LE: Choose θ that maximizes the probability of observed data MLE: MLE:Choose Choose θthat thatmaximizes maximizesthe theprobability probability of of observed observed data data Independent draws Identically distributed slide by Barnabás Póczos & Alex Smola That s exactly the Frequency of heads

15 Maximum Maximum Likelihood Likelihood Estimation Maximum Estimation Likelihood Estimation Estimation LE: Choose θ that maximizes the probability of observed data MLE: MLE:Choose Choose θthat thatmaximizes maximizesthe theprobability probability of of observed observed data data Independent draws Identically distributed slide by Barnabás Póczos & Alex Smola That s exactly the Frequency of heads

16 Maximum Maximum Likelihood Likelihood Estimation Maximum Estimation Likelihood Estimation Estimation LE: Choose θ that maximizes the probability of observed data MLE: MLE:Choose Choose θthat thatmaximizes maximizesthe theprobability probability of of observed observed data data Independent draws Identically distributed slide by Barnabás Póczos & Alex Smola That s exactly the Frequency of heads

17 Question (2) How good is this MLE estimation??? slide by Barnabás Póczos & Alex Smola 17

18 How many flips do I need? I flipped the coins 5 times: 3 heads, 2 tails What if I flipped 30 heads and 20 tails? slide by Barnabás Póczos & Alex Smola Which estimator should we trust more? The more the merrier??? 18

19 Simple bound Let θ* be the true parameter. For n = αh+αt, and For any ε>0: Hoeffding s inequality: slide by Barnabás Póczos & Alex Smola 19

20 Probably Approximate Correct (PAC) Learning I want to know the coin parameter θ, within ε = 0.1 error with probability at least 1-δ = How many flips do I need? slide by Barnabás Póczos & Alex Smola Sample complexity: 20

21 Question (3) Why is this a machine learning problem??? improve their performance (accuracy of the predicted prob. ) at some task (predicting the probability of heads) with experience (the more coins we flip the better we are) slide by Barnabás Póczos & Alex Smola 21

22 What continuous What about about continuous features? features? Let us try Gaussians slide by Barnabás Póczos & Alex Smola σ2 µ=0 σ2 µ=

23 MLE for Gaussian mean and variance Choose θ= (µ,σ 2 ) that maximizes the probability of observed data Independent draws Identically distributed slide by Barnabás Póczos & Alex Smola 23

24 MLE mean MLEfor forgaussian Gaussian mean and variance variance and slide by Barnabás Póczos & Alex Smola Note: MLE for the variance of a Gaussian is biased Note: MLE for result the variance Gaussian is biased [Expected of estimationofis a not the true parameter!] [Expected result of estimation is not the true parameter!] Unbiased variance estimator: Unbiased variance estimator: 24 27

25 What about prior knowledge? (MAP Estimation) 25

26 What about prior knowledge? We know the coin is close to What can we do now? The Bayesian way Rather than estimating a single θ, we obtain a distribution over possible values of θ Before data After data

27 Prior distribution What prior? What distribution do we want for a prior? Represents expert knowledge (philosophical approach) Simple posterior form (engineer s approach) Uninformative priors: Uniform distribution Conjugate priors: Closed-form representation of posterior P(θ) and P(θ D) have the same form 27

28 In order to proceed we will need: Bayes Rule 28

29 Chain Rule Rule & & Bayes Bayes Rule Chain Rule Chain rule: Bayes rule: Bayes ruleisisimportant important for Bayes rule forreverse reverseconditioning. conditioning

30 Bayesian Learning Use Bayes rule: Or equivalently: posterior likelihood prior 30

31 MAP estimation for Binomial distribution Coin flip problem Likelihood is Binomial If the prior is Beta distribution, ) posterior is Beta distribution P( ) and P( D) have the same form! [Conjugate prior] 31

32 Beta distribution More concentrated as values of α, β increase 32

33 Beta conjugate prior As n = α H + αt increases As we get more samples, effect of prior is washed out 33

34 Han Solo and Bayesian Priors C3PO: Sir, the possibility of successfully navigating an asteroid field is approximately 3,720 to 1! Han: Never tell me the odds! 34

35 MLE vs. MAP! Maximum Likelihood estimation (MLE) Choose value that maximizes the probability of observed data! Maximum a posteriori (MAP) estimation Choose value that is most probable given observed data and prior belief When is MAP same as MLE? 35

36 From Binomial to Multinomial Example: Dice roll problem (6 outcomes instead of 2) Likelihood is ~ Multinomial(θ = {θ 1, θ 2,..., θ k }) If prior is Dirichlet distribution, et distribution, Then posterior is Dirichlet distribution For Multinomial, conjugate prior is Dirichlet distribution. 36

37 Bayesians vs. Frequentists You are no good when sample is small You give a different answer for different priors 37

38 Recap: What about prior knowledge? (MAP Estimation) 38

39 Recap: What about prior knowledge? We know the coin is close to What can we do now? The Bayesian way Rather than estimating a single θ, we obtain a distribution over possible values of θ Before data After data

40 Recap: Chain Rule & Bayes Rule Chain rule: Bayes rule: 40

41 Recap: Bayesian Learning D is the measured data Our goal is to estimate parameter θ Use Bayes rule: Or equivalently: posterior likelihood prior 41

42 Recap: MAP estimation for Binomial distribution In the coin flip problem: Likelihood is Binomial: If the prior is Beta: then the posterior is Beta distribution 42

43 Recap: Beta conjugate prior As n = α H + αt increases As we get more samples, effect of prior is washed out 43

44 Application of Bayes Rule 44

45 AIDS test (Bayes rule) Data Approximately 0.1% are infected Test detects all infections Test reports positive for 1% healthy people Probability of having AIDS if test is positive Only 9%!

46 Improving the diagnosis Use a weaker follow-up test! Approximately 0.1% are infected Test 2 reports positive for 90% infections Test 2 reports positive for 5% healthy people = 64%!

47 AIDS test (Bayes rule) Why can t we use Test 1 twice? Outcomes are not independent, but tests 1 and 2 conditionally independent (by assumption): 47

48 The Naïve Bayes Classifier 48

49 Data for spam filtering date time recipient path IP number sender encoding many more features R Delivered-To: alex.smola@gmail.com Received: by with SMTP id s51cs361171web; Tue, 3 Jan :17: (PST) Received: by with SMTP id s17mr eba ; Tue, 03 Jan :17: (PST) Return-Path: <alex+caf_=alex.smola=gmail.com@smola.org> Received: from mail-ey0-f175.google.com (mail-ey0-f175.google.com [ ]) by mx.google.com with ESMTPS id n4si eef (version=tlsv1/sslv3 cipher=other); Tue, 03 Jan :17: (PST) Received-SPF: neutral (google.com: is neither permitted nor denied by best guess record for domain of alex+caf_=alex.smola=gmail.com@smola.org) clientip= ; Authentication-Results: mx.google.com; spf=neutral (google.com: is neither permitted nor denied by best guess record for domain of alex +caf_=alex.smola=gmail.com@smola.org) smtp.mail=alex+caf_=alex.smola=gmail.com@smola.org; dkim=pass (test mode) header.i=@googl .com Received: by eaal1 with SMTP id l1so eaa.6 for <alex.smola@gmail.com>; Tue, 03 Jan :17: (PST) Received: by with SMTP id ie18mr bkc ; Tue, 03 Jan :17: (PST) X-Forwarded-To: alex.smola@gmail.com X-Forwarded-For: alex@smola.org alex.smola@gmail.com Delivered-To: alex@smola.org Received: by with SMTP id k6cs206093bki; Tue, 3 Jan :17: (PST) Received: by with SMTP id bh19mr vdb ; Tue, 03 Jan :17: (PST) Return-Path: <althoff.tim@googl .com> Received: from mail-vx0-f179.google.com (mail-vx0-f179.google.com [ ]) by mx.google.com with ESMTPS id dt4si vdb (version=tlsv1/sslv3 cipher=other); Tue, 03 Jan :17: (PST) Received-SPF: pass (google.com: domain of althoff.tim@googl .com designates as permitted sender) client-ip= ; Received: by vcbf13 with SMTP id f13so vcb.10 for <alex@smola.org>; Tue, 03 Jan :17: (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=googl .com; s=gamma; h=mime-version:sender:date:x-google-sender-auth:message-id:subject :from:to:content-type; bh=wcbdz5sxac25dph02xcrydodts993hkwsavxpgrfh0w=; b=wk2b2+exwnf/gvtkw6uuvkup4xeoknljq3usytm0rark8dsfjyoqsiheap9yssxp6o 7ngGoTzYqd+ZsyJfvQcLAWp1PCJhG8AMcnqWkx0NMeoFvIp2HQooZwxSOCx5ZRgY+7qX uibbdna4ludxj6ufe16spldckptd8oz3gr7+o= MIME-Version: 1.0 Received: by with SMTP id e17mr vcp ; Tue, 03 Jan :17: (PST) Sender: althoff.tim@googl .com Received: by with HTTP; Tue, 3 Jan :17: (PST) Date: Tue, 3 Jan :17: X-Google-Sender-Auth: 6bwi6D17HjZIkxOEol38NZzyeHs Message-ID: <CAFJJHDGPBW+SdZg0MdAABiAKydDk9tpeMoDijYGjoGO-WC7osg@mail.gmail.com> Subject: CS 281B. Advanced Topics in Learning and Decision Making

50 Naïve Bayes Assumption Naïve Bayes assumption: Features X 1 and X 2 are conditionally independent given the class label Y: More generally: 50

51 Naïve Bayes Assumption, Example Task: Predict whether or not a picnic spot is enjoyable Training Data: X = (X 1 X 2 X 3 X d ) Y n rows Naïve Bayes assumption: How many parameters to estimate? (X is composed of d binary features, Y has K possible class labels) (2 d -1)K vs (2-1)dK 1 51

52 Naïve Bayes Classifier Given: Class prior P(Y) d conditionally independent features X 1, X d given the class label Y For each X i feature, we have the conditional likelihood P(X i Y) Naïve Bayes Decision rule: 52

53 Naïve Bayes Algorithm for discrete features Training data: n d-dimensional discrete features + K class labels We need to estimate these probabilities! Estimate them with MLE (Relative Frequencies)! 53

54 Naïve Bayes Algorithm for discrete features We need to estimate these probabilities! Estimators For Class Prior For Likelihood NB Prediction for test data: 19 54

55 Subtlety: Insufficient training data For example, What now??? 21 55

56 Naïve Bayes Alg Discrete features Training data: Use your expert knowledge & apply prior distributions: Add m virtual examples Same as assuming conjugate priors Assume priors: MAP Estimate: called Laplace smoothing # virtual examples with Y = b 56 22

57 Case Study: Text Classification 57

58 Is this spam? 58

59 Positive or negative movie review? unbelievably disappointing Full of zany characters and richly applied satire, and some great plot twists this is the greatest screwball comedy ever filmed It was pathetic. The worst part about it was the boxing scenes. slide by Dan Jurafsky 59

60 What is the subject of this article? MEDLINE Article MeSH Subject Category Hierarchy Antogonists and Inhibitors? Blood Supply Chemistry slide by Dan Jurafsky Drug Therapy Embryology Epidemiology 60

61 Text Classification Assigning subject categories, topics, or genres Spam detection Authorship identification Age/gender identification Language Identification Sentiment analysis slide by Dan Jurafsky 61

62 Text Classification: definition Input: - a document d - a fixed set of classes C = {c1, c2,, cj} Output: a predicted class c C slide by Dan Jurafsky 62

63 Hand-coded rules Rules based on combinations of words or other features - spam: black-list-address OR ( dollars AND have been selected ) Accuracy can be high - If rules carefully refined by expert But building and maintaining these rules is expensive slide by Dan Jurafsky 63

64 Text Classification and Naive Bayes Classify s - Y = {Spam, NotSpam} Classify news articles - Y = {what is the topic of the article?} What are the features X? The text! Let X i represent i th word in the document 64

65 X i represents i th word in document 65

66 NB for Text Classification A problem: The support of P(X Y) is huge! Article at least 1000 words, X={X 1,,X 1000 } X i represents i th word in document, i.e., the domain of X i is the entire vocabulary, e.g., Webster Dictionary (or more). X i 2 {1,,50000} ) K( ) parameters to estimate without the NB assumption. 66

67 NB for Text Classification X i 2 {1,,50000} ) K( ) parameters to estimate. NB assumption helps a lot!!! If P(X i =x i Y=y) is the probability of observing word x i at the i th position in a document on topic y ) 1000K( ) parameters to estimate with NB assumption NB assumption helps, but still lots of parameters to estimate. 6726

68 Bag of words model Typical additional assumption: Position in document doesn t matter: P(X i =x i Y=y) = P(X k =x i Y=y) Bag of words model order of words on the page ignored The document is just a bag of words: i.i.d. words Sounds really silly, but often works very well! ) K( ) parameters to estimate The probability of a document with words x 1,x 2, 68 27

69 The bag of words representation I love this movie! It's sweet, but with satirical humor. The dialogue is great and the adventure scenes are fun It manages to be whimsical and romantic while laughing at the conventions of the fairy tale genre. I would recommend it to just about anyone. I've seen it several times, and I'm always happy to see it again whenever I have a friend who hasn't seen it yet. γ( )=c slide by Dan Jurafsky 69

70 The bag of words representation γ( I love this movie! It's sweet, but with satirical humor. The dialogue is great and the adventure scenes are fun It manages to be whimsical and romantic while laughing at the conventions of the fairy tale genre. I would recommend it to just about anyone. I've seen it several times, and I'm always happy to see it again whenever I have a friend who hasn't seen it yet. )=c slide by Dan Jurafsky 70

71 The bag of words representation: using a subset of words γ( x love xxxxxxxxxxxxxxxx sweet xxxxxxx satirical xxxxxxxxxx xxxxxxxxxxx great xxxxxxx xxxxxxxxxxxxxxxxxxx fun xxxx xxxxxxxxxxxxx whimsical xxxx romantic xxxx laughing xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx xxxxxxxxxxxxx recommend xxxxx xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx x several xxxxxxxxxxxxxxxxx xxxxx happy xxxxxxxxx again xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx xxxxxxxxxxxxxxxx )=c slide by Dan Jurafsky 71

72 The bag of words representation great 2 γ( love 2 recommend 1 laugh 1 happy )=c slide by Dan Jurafsky 72

73 Doc Words Class ˆP(c) = N c N ˆP(w c) = count(w,c)+1 count(c)+ V Training 1 Chinese Beijing Chinese c 2 Chinese Chinese Shanghai c 3 Chinese Macao c 4 Tokyo Japan Chinese j Test 5 Chinese Chinese Chinese Tokyo Japan? slide by Dan Jurafsky Priors: P(c)= 3 4 P(j)= 1 4 Choosing a class: P(c d5) 3/4 * (3/7) 3 * 1/14 * 1/ Conditional Probabilities: P(Chinese c) = (5+1) / (8+6) = 6/14 = 3/7 P(Tokyo c) = (0+1) / (8+6) = 1/14 P(j d5) 1/4 * (2/9) 3 * 2/9 * 2/9 P(Japan c) = (0+1) / (8+6) = 1/ P(Chinese j) = (1+1) / (3+6) = 2/9 P(Tokyo j) = (1+1) / (3+6) = 2/9 P(Japan j) = (1+1) / (3+6) = 2/9 73

74 What if features are continuous? e.g., character recognition: Xi is intensity at i th pixel i Gaussian Naïve Bayes (GNB): Naïve Bayes (GNB): Different mean and variance for each class k and each pixel i. Sometimes assume variance is independent of Y (i.e., σi), or independent of Xi (i.e., σk) or both (i.e., σ) 74

75 Estimating parameters: Y discrete, X i continuous 75

76 Estimating parameters: Y discrete, X i continuous Maximum likelihood estimates: i th pixel in j th training image k th class j th training image 76

77 Twenty news groups results Naïve Bayes: 89% accuracy 77

78 Case Study: Classifying Mental States 78

79 Example: GNB for classifying mental states ~1 mm resolution ~2 images per sec. 15,000 voxels/image non-invasive, safe measures Blood Oxygen Level Dependent (BOLD) response [Mitchell et al.] 79

80 Brain scans can track activation with precision and sensitivity

81 Learned Naïve Bayes Models Means for P(BrainActivity WordCategory) Tool words Pairwise classification accuracy: 78-99%, 12 participants Building Building words [Mitchell et al.] 81

82 What you should know Naïve Bayes classifier What s the assumption Why we use it How do we learn it Why is Bayesian (MAP) estimation important Text classification Bag of words model Gaussian NB Features are still conditionally independent Each feature has a Gaussian distribution given class 82