CS 109 Lecture 23 May 18th, 2016

Transcription

1 CS 109 Lecture 23 May 18th, 2016

2 New Datasets Heart Ancestry Netflix

3 Our Path Parameter Estimatin

4 Machine Learning: Frmally Many different frms f Machine Learning We fcus n the prblem f predictin Want t make a predictin based n bservatins Vectr X f m bserved variables: <X 1, X 2,, X m > X 1, X 2,, X m are called input features/variables Based n bserved X, want t predict unseen variable called utput feature/variable (r the dependent variable ) Seek t learn a functin g(x) t predict : ˆ = g( X ) When is discrete, predictin f is called classificatin When is cntinuus, predictin f is called regressin

5 A (Very Shrt) List f Applicatins Machine learning widely used in many cntexts Stck price predictin Using ecnmic indicatrs, predict if stck will g up/dwn Cmputatinal bilgy and medical diagnsis Predicting gene expressin based n DNA Determine likelihd fr cancer using clinical/demgraphic data Predict peple likely t purchase prduct r click n ad Based n past purchases, yu might want t buy Credit card fraud and telephne fraud detectin Based n past purchases/phne calls is a new ne fraudulent? Saves cmpanies billins(!) f dllars annually Spam detectin (gmail, htmail, many thers)

6 That list is ridiculusly shrt J

7 Mtivating Example

8 What is Bayes Ding in my Mail Server This is spam: Let s get Bayesian n yur spam: Cntent analysis details: (49.5 hits, 7.0 required) 0.9 RCVD_IN_PBL RBL: Received via a relay in Spamhaus PBL [ listed in zen.spamhaus.rg] 1.5 URIBL_WS_SURBL Cntains an URL listed in the WS SURBL blcklist [URIs: recragas.cn] 5.0 URIBL_JP_SURBL Cntains an URL listed in the JP SURBL blcklist [URIs: recragas.cn] 5.0 URIBL_OB_SURBL Cntains an URL listed in the OB SURBL blcklist [URIs: recragas.cn] 5.0 URIBL_SC_SURBL Cntains an URL listed in the SC SURBL blcklist [URIs: recragas.cn] 2.0 URIBL_BLACK Cntains an URL listed in the URIBL blacklist [URIs: recragas.cn] 8.0 BAES_99 BOD: Bayesian spam prbability is 99 t 100% [scre: ] Wh was crazy enugh t think f that?

9 Spam, Spam G Away! The cnstant battle with spam And machine-learning algrithms develped t merge and rank large sets f Ggle search results allw us t cmbine hundreds f factrs t classify spam. Surce:

10 Training a Learning Machine We cnsider statistical learning paradigm here We are given set f N training instances Each training instance is pair: (<x 1, x 2,, x m >, y) Training instances are previusly bserved data Gives the utput value y assciated with each bserved vectr f input values <x 1, x 2,, x m > Learning: use training data t specify g(x) Generally, first select a parametric frm fr g(x) Then, estimate parameters f mdel g(x) using training data Fr regressin, usually want g(x) that minimizes E[( g(x)) 2 ] Mean squared errr (MSE) lss functin. (Others exist.) Fr classificatin, generally best chice f g( X ) = arg max Pˆ( X ) y

11 The Machine Learning Prcess Learning algrithm g(x) (Classifier) Output (Class) Training data Training data: set f N pre-classified data instances X N training pairs: (<x> (1),y (1) ), (<x> (2),y (2) ),, (<x> (N), y (N) ) Use superscripts t dente i-th training instance Learning algrithm: methd fr determining g(x) Given a new input bservatin f X = <X 1, X 2,, X m > Use g(x) t cmpute a crrespnding utput (predictin) When predictin is discrete, we call g(x) a classifier and call the utput the predicted class f the input

12 Training Real Wrld Prblem Mdel the prblem Frmal Mdel θ Training Data Testing Data Learning Algrithm g(θ * ) Classificatin Accuracy

13 Testing Real Wrld Prblem Mdel the prblem Frmal Mdel θ Training Data Testing Data Learning Algrithm g(θ * ) Classificatin Accuracy

14 Linear Regressin

15 A Grunding Example: Linear Regressin Predict real value based n bserving variable X Assume mdel is linear: ˆ = g( X ) = ax + b Training data Each vectr X has ne bserved variable: <X 1 > (just call it X) is cntinuus utput variable Given N training pairs: (<x> (1),y (1) ), (<x> (2),y (2) ),, (<x> (N), y (N) ) Use superscripts t dente i-th training instance Determine a and b by minimizing E[( g(x)) 2 ]

16 Predicting CO 2 X 1 = Temperature X 2 = Elevatin X 3 = CO 2 level yesterday X 4 = GDP f regin X 5 = Acres f frest grwth = CO 2 levels

17 Hw Did We Get Linear Regressin? N training pairs: (x (1),y (1) ), (x (2),y (2) ),, (x (N), y (N) ) 1. Linear Regressin Mdel: = 1 X X n 1 X n 1 + n 1+Z = T X + Z Z N(0, 2 ) 2. Find the LL functin and chse thetas which maximize it ˆ MLE = argmax nx ( (i) T x (i) ) 2 i=1 3. Use an ptimizer t calculate each theta.

18 Classificatin

19 A Simple Classificatin Example Predict based n bserving variables X X has discrete value frm {1, 2, 3, 4} X dentes temperature range tday: <50, 50-60, 60-70, >70 has discrete value frm {rain, sun} dentes general weather utlk tmrrw px, ( x, y) p P( X ) = = p ( x) Nte Bayes Thm.: Fr new X, predict ˆ = g( X) = Nte p x (x) is nt affected by chice f y, yielding: X X ( x p arg max Pˆ( X ) y X y) p ( x) ( y) ˆ = g( X) = arg max y Pˆ( X ) = arg max y Pˆ( X, ) = arg max y Pˆ( X ) Pˆ( )

20 Brute Frce Classificatin

21 Estimating the Cmplete Jint Frm last slide: ˆ = g( X) = arg max y Pˆ( X ) = arg max y Pˆ( X, ) = arg max y Pˆ( X ) P First idea: Let (X,) be ne giant multinmial! Say X can take n the values 1, 2, 3, 4 and can take n the values 1,2 X θ 1,1 θ 1,2 θ 1,3 θ 1,4 2 θ 2,1 θ 2,2 θ 2,3 θ 2,4 Estimate these and use them t make ur predictin

22 Estimating the Cmplete Jint Given training data, cmpute jint PMF: p X, (x, y) X MLE: cunt number f times each pair (x, y) appears MAP using Laplace prir: add 1 t all the MLE cunts Nrmalize t get true distributin (sums t 1) Observed 50 data pints: ˆ = p MLE cunt in cell ttal # data pints MLE estimate p (y) rain sun p X (x) X rain sun ˆ p Laplace X cunt in cell+ 1 = ttal # data pints + ttal # cells Laplace (MAP) estimate p (y) rain sun p X (x)

23 Classify New Observatins Say tday s temperature is 75, s X = 4 Recall X temperature ranges: <50, 50-60, 60-70, >70 Predictin fr (weather utlk tmrrw) ) = arg max Pˆ( X ) Pˆ( ) ˆ = arg max Pˆ( X, y MLE estimate X p (y) y Laplace (MAP) estimate X p (y) rain sun p X (x) rain sun p X (x) What if we asked what is prbability f rain tmrrw? MLE: abslutely, psitively n chance f rain! Laplace estimate: small chance à never say never

24 Classificatin with Multiple Observatins Say, we have m input values X = <X 1, X 2,, X m > Nte that variables X 1, X 2,, X m can be dependent! In thery, culd predict as befre, using ˆ = arg max Pˆ( X, ) = arg max Pˆ( X ) Pˆ( ) y Why wn t this necessarily wrk in practice? Need t estimate P(X 1, X 2,, X m ) Fine if m is small, but what if m = 10 r 100 r 10,000? y Need ridiculus amunt f data fr gd prbability estimates! Likely t have many 0 s in table (bad times) Need t cnsider a simpler mdel

25 And Learn

26 Netflix and Learn Say, we have m input values X = <X 1, X 2,, X m > and a single. Each X i represents if a user liked mvie i. Let s think abut the jint distributin fr different values f m

27 Netflix and Learn: m = 1 Say, we have m input values X = <X 1, X 2,, X m > and a single. Each X i represents if a user liked mvie i. X θ 0,0 θ 0,1 1 θ 1,0 θ 1,1

28 Netflix and Learn: m = 2 Say, we have m input values X = <X 1, X 2,, X m > and a single. Each X i represents if a user liked mvie i. X 2 = 0 X 2 = 1 X X θ 0,0,0 θ 0,1,0 0 θ 0,0,1 θ 0,1,1 1 θ 1,0,0 θ 1,1,0 1 θ 1,0,1 θ 1,1,1

29 Netflix and Learn: m = 3 Say, we have m input values X = <X 1, X 2,, X m > and a single. Each X i represents if a user liked mvie i. X 2 = 0 X 2 = 1 X X X 3 = 0 0 θ 0,0,0,0 θ 0,1,0,0 0 θ 0,0,1,0 θ 0,1,1,0 1 θ 1,0,0,0 θ 1,1,0,0 1 θ 1,0,1,0 θ 1,1,1,0 X 2 = 0 X 2 = 1 X 3 = 1 X θ 0,0,0,1 θ 0,1,0,1 X θ 0,0,1,1 θ 0,1,1,1 1 θ 1,0,0,1 θ 1,1,0,1 1 θ 1,0,1,1 θ 1,1,1,1

30 And if m=100?

31 What is the big O fr # parameters? m = # features.

32 Big O f Brute Frce Jint What is the big O fr # parameters? m = # features. O(2 n )

33 Nt ging t cut it!

34 Naïve Bayes Classifier Say, we have m input values X = <X 1, X 2,, X m > Assume variables X 1, X 2,, X m are cnditinally independent given Really dn t believe X 1, X 2,, X m are cnditinally independent Just an apprximatin we make t be able t make predictins This is called the Naive Bayes assumptin, hence the name Predict using P(X ) But, we nw have: by cnditinal independence Nte: cmputatin f PMF table is linear in m : O(m) = P( X ˆ = arg max P( X, ) = m 1, X 2,... X m ) = P( X i ) i= 1 y arg max P( X Dn t need much data t get gd prbability estimates y ) P( )

35 Naïve Bayes Example Predict based n bserving variables X 1 and X 2 X 1 X 1 and X 2 are bth indicatr variables X 1 dentes likes Star Wars, X 2 dentes likes Harry Ptter is indicatr variable: likes Lrd f the Rings 0 1 Use training data t estimate PMFs: MLE estimates X pˆ (, ), ˆ, x y p ( y) Say smene likes Star Wars (X 1 = 1), but nt Harry Ptter (X 2 = 0) Will they like Lrd f the Rings? Need t predict : ˆ = arg max Pˆ( X ) Pˆ( ) = y X i MLE estimates arg max Pˆ( X y i ) Pˆ( X # MLE est ) Pˆ( 1 2 )

36 One SciFi/Fantasy t Rule them All X 1 Predictin fr is value f maximizing P(X, ): 0 1 MLE estimates X 2 ˆ = arg max Pˆ( X ) Pˆ( ) = y 0 1 MLE estimates arg max Pˆ( X ) Pˆ( X # Since P(X, =1) > P(X, =0), we predict Ŷ = 1 y MLE est ) Pˆ( 1 2 Cmpute P(X, =0): Pˆ( X = 1 = 0) Pˆ( X 2 = 0 = 0) Pˆ( Pˆ( X = 1, = 0) Pˆ( X 2 = 0, = 0) = ˆ( 0) ˆ( 1 P = P = 0) Pˆ( = 0) = Cmpute P(X, =1): Pˆ( X = 1 = 1) Pˆ( X = 0 = 1) Pˆ( Pˆ( X = 1, = 1) Pˆ( X 2 = 0, = 1) = ˆ( 1) ˆ( 1 P = P = 1) Pˆ( = 1) = ) 1) 0)

37 Classificatin Want t predict if an is spam r nt Start with the input data Cnsider a lexicn f m wrds (Nte: in English m 100,000) Define m indicatr variables X = <X 1, X 2,, X m > Each variable X i dentes if wrd i appeared in a dcument r nt Nte: m is huge, s make Naive Bayes assumptin Define utput classes t be: {spam, nn-spam} Given training set f N previus s Fr each message, we have a training instance: X = <X 1, X 2,, X m > nting fr each wrd, if it appeared in Each message is als marked as spam r nt (value f )

38 Training the Classifier Given N training pairs: (<x> (1),y (1) ), (<x> (2),y (2) ),, (<x> (N), y (N) ) Learning Estimate prbabilities P() and each P(X i ) fr all i Many wrds are likely t nt appear at all in given set f Laplace estimate: Classificatin Fr a new , generate X = <X 1, X 2,, X m > Classify as spam r nt using: ˆ = arg max Pˆ( X ) Pˆ( ) pˆ( X i = 1 = spam) Emply Naive Bayes assumptin: Pˆ ( X ) = Laplace (#spam s with wrd i) + 1 = ttal # spam s + 2 y m i= 1 Pˆ( X i )

39 Hw Des This D? After training, can test with anther set f data Testing set als has knwn values fr, s we can see hw ften we were right/wrng in predictins fr Spam data data set: s (1578 spam, 211 nn-spam) First, messages (by time) used fr training Next 251 messages used t test learned classifier Criteria: Precisin = # crrectly predicted class / # predicted class Recall = # crrectly predicted class / # real class messages Spam Nn-spam Precisin Recall Precisin Recall Wrds nly 97.1% 94.3% 87.7% 93.4% Wrds + add l features 100% 98.3% 96.2% 100%

40 On biased datasets

41 Ethics and Datasets? Smetimes machine learning feels universally unbiased. We can even prve ur estimatrs are unbiased J Ggle/Nikn/HP had biased datasets

42 Ancestry dataset predictin East Asian r Ad Mixed American (Native, Eurpean and African Americans)

43 Is the ancestry dataset biased?

44 es!

45 It is much easier t write a binary classifier when learning ML fr the first time

46 Learn Tw Things Frm This 1. What classificatin with DNA Single Nucletide Plymrphisms lks like. 2. That genetic ancestry paints a mre realistic picture f hw we are mixed in many nuanced ways. 3. The imprtance f chsing the right data t learn frm. ur results will be as biased as yur dataset. Knw it s yu can beat it!

47 Ethics in Machine Learning is a whle new field