Classification Bayesian Classifiers

Transcription

1 lassfcaton Bayesan lassfers Jeff Howbert Introducton to Machne Learnng Wnter

2 Bayesan classfcaton A robablstc framework for solvng classfcaton roblems. Used where class assgnment s not determnstc,.e. a artcular set of attrbute values wll sometmes be assocated wth one class, sometmes wth another. Requres estmaton of osteror robablty for each class, gven a set of attrbute values: 1, 2, L, n for each class Then use decson theory to make redctons for a new samle Jeff Howbert Introducton to Machne Learnng Wnter

3 Jeff Howbert Introducton to Machne Learnng Wnter ondtonal robablty: Bayes theorem: Bayesan classfcaton,, osteror robablty lkelhood ror robablty evdence

4 Gven: Eamle of Bayes theorem A doctor knows that menngts causes stff neck 50% of the tme Pror robablty of any atent havng menngts s 1/50,000 Pror robablty of any atent havng stff neck s 1/20 If a atent has stff neck, what s the robablty he/she has menngts? S M M 0.5 1/ M S S 1/ Jeff Howbert Introducton to Machne Learnng Wnter

5 Bayesan classfers Treat each attrbute and class label as random varables. Gven a samle wth attrbutes 1, 2,, n : Goal s to redct class. Secfcally, we want to fnd the value of that mamzes 1, 2,, n. an we estmate 1, 2,, n drectly from data? Jeff Howbert Introducton to Machne Learnng Wnter

6 Aroach: Bayesan classfers omute the osteror robablty 1, 2,, n for each value of usng Bayes theorem: 1, 2, K, hoose value of that mamzes 1, 2,, n, 2, K,, Equvalent to choosng value of that mamzes 1, 2,, n We can gnore denomnator why? n n, K, Easy to estmate rors from data. How? The real challenge: how to estmate 1, 2,, n? Jeff Howbert Introducton to Machne Learnng Wnter n

7 Bayesan classfers How to estmate 1, 2,, n? In the general case, where the attrbutes j have deendences, ths requres estmatng the full jont dstrbuton 1, 2,, n for each class. There s almost never enough data to confdently make such estmates. Jeff Howbert Introducton to Machne Learnng Wnter

8 Naïve Bayes classfer Assume ndeendence among attrbutes j when class s gven: 1, 2,, n 1 2 n Usually straghtforward and ractcal to estmate j for all j and. New samle s classfed to f Π j s mamal. Jeff Howbert Introducton to Machne Learnng Wnter

9 10 How to estmate j from data? c c Td Refund Martal Status c Taable Income 1 Yes Sngle 125K No 2 No Marred 100K No 3 No Sngle 70K No 4 Yes Marred 120K No Evade 5 No Dvorced 95K Yes 6 No Marred 60K No 7 Yes Dvorced 220K No 8 No Sngle 85K Yes 9 No Marred 75K No 10 No Sngle 90K Yes lass rors: N / N No 7/10 Yes 3/10 For dscrete attrbutes: j j / N where j s number of nstances n class havng attrbute value j Eamles: Status Marred No 4/7 Refund Yes Yes 0 Jeff Howbert Introducton to Machne Learnng Wnter

10 How to estmate j from data? For contnuous attrbutes: Dscretze the range nto bns relace wth an ordnal attrbute Two-way slt: < v or > v relace wth a bnary attrbute Probablty densty estmaton: assume attrbute follows some standard arametrc robablty dstrbuton usually a Gaussan use data to estmate arameters of dstrbuton e.g. mean and varance once dstrbuton s known, can use t to estmate the condtonal robablty j Jeff Howbert Introducton to Machne Learnng Wnter

11 10 How to estmate j from data? Td Refund Martal Status Taable Income 1 Yes Sngle 125K No 2 No Marred 100K No 3 No Sngle 70K No 4 Yes Marred 120K No Evade 5 No Dvorced 95K Yes 6 No Marred 60K No 7 Yes Dvorced 220K No 8 No Sngle 85K Yes 9 No Marred 75K No 10 No Sngle 90K Yes Gaussan dstrbuton: P j 2πσ 2σ one for each j, ar For Income lass No : 2 j samle mean 110 samle varance e j μ j 2 j 2 Income 120 No 1 2π e Jeff Howbert Introducton to Machne Learnng Wnter

12 Eamle of usng naïve Bayes classfer Gven a Test Record: Refund No, Status Marred, Income 120K Refund Yes No 3/7 Refund No No 4/7 Refund Yes Yes 0/3 Refund No Yes 3/3 Martal Status Sngle No 2/7 Martal Status Dvorced No 1/7 Martal Status Marred No 4/7 Martal Status Sngle Yes 2/3 Martal Status Dvorced Yes 1/3 Martal Status Marred Yes 0/3 For Taable Income: If lass No: samle mean 110 samle varance 2975 If lass Yes: samle mean 90 samle varance 25 lass No Refund No lass No Marred lass No Income 120K lass No 4/7 4/ lass Yes Refund No lass Yes Marred lass Yes Income 120K lass Yes No No > Yes Yes therefore No > Yes > lass No Jeff Howbert Introducton to Machne Learnng Wnter

13 Naïve Bayes classfer Problem: f one of the condtonal robabltes s zero, then the entre eresson becomes zero. Ths s a sgnfcant ractcal roblem, esecally when tranng samles are lmted. Ways to mrove robablty estmaton: Orgnal : Lalace: m - estmate : j j j N N N N j j c N Jeff Howbert Introducton to Machne Learnng Wnter j N + + m m c: number of classes : ror robablty m: arameter

14 Eamle of Naïve Bayes classfer Name Gve Brth an Fly Lve n Water Have Legs lass human yes no no yes mammals ython no no no no non-mammals salmon no no yes no non-mammals whale yes no yes no mammals frog no no sometmes yes non-mammals komodo no no no yes non-mammals bat yes yes no yes mammals geon no yes no yes non-mammals cat yes no no yes mammals leoard shark yes no yes no non-mammals turtle no no sometmes yes non-mammals engun no no sometmes yes non-mammals orcune yes no no yes mammals eel no no yes no non-mammals salamander no no sometmes yes non-mammals gla monster no no no yes non-mammals latyus no no no yes mammals owl no yes no yes non-mammals dolhn yes no yes no mammals eagle no yes no yes non-mammals X X X X X: attrbutes M: class mammal N: class non-mammal M N M M N N Gve Brth an Fly Lve n Water Have Legs lass yes no yes no? X M M > X N N > mammal Jeff Howbert Introducton to Machne Learnng Wnter

15 Summary of naïve Bayes Robust to solated nose samles. Handles mssng values by gnorng the samle durng robablty estmate calculatons. Robust to rrelevant attrbutes. NOT robust to redundant attrbutes. Indeendence assumton does not hold n ths case. Use other technques such as Bayesan Belef Networks BBN. Jeff Howbert Introducton to Machne Learnng Wnter