Classification learning II

Transcription

1 Lecture 8 Classfcaton learnng II Mlos Hauskrecht mlos@cs.ptt.edu 539 Sennott Square Logstc regresson model Defnes a lnear decson boundar Dscrmnant functons: g g g g here g z / e z f, g g - s a logstc functon Input vector z f, d Logstc functon d

2 Logstc regresson model. Decson boundar LR defnes a lnear decson boundar Eample: classes blue and red ponts Decson boundar Logstc regresson: parameter learnng Log lkelhood Dervatves of the loglkelhood Gradent descent: log log, n D l,, n n f g D l ], [ k D l k k k Nonlnear n eghts!! n k k k f k ], [,, n j j z g D l

3 Generatve approach to classfcaton Logstc regresson: learns a model of p Generatve approach:. Represents and learns the jont dstrbuton p,. Uses t to defne probablstc dscrmnant functons E.g. g o p g p pcal jont model p, p p p = Class-condtonal dstrbutons denstes bnar classfcaton: to class-condtonal dstrbutons p p p = Prors on classes - probablt of class bnar classfcaton: Bernoull dstrbuton p p Quadratc dscrmnant analss QDA Model: Class-condtonal dstrbutons multvarate normal dstrbutons N μ, Σ for p μ, Σ ~ ~ N μ, Σ for Multvarate normal ~ N μ, Σ d / Σ / ep μ Σ μ Prors on classes class, Bernoull dstrbuton p, ~ Bernoull {,} 3

4 Learnng Quadratc dscrmnant analss QDA Learnng Class-condtonal dstrbutons Learn parameters of multvarate normal dstrbutons N μ, Σ for N μ, Σ for ~ ~ Use the denst estmaton methods Learnng Prors on classes class, ~ Bernoull Learn the parameter of the Bernoull dstrbuton Agan use the denst estmaton methods p, {,} QDA.5.5 g g

5 Gaussan class-condtonal denstes. QDA: Makng class decson Bascall e need to desgn dscrmnant functons Posteror of a class choose the class th better posteror probablt p p then = g g else = p p, Σ p, Σ p p p, Σ It s suffcent to compare: p, Σ p p, Σ p p 5

6 QDA: Quadratc decson boundar Contours of class-condtonal denstes QDA: Quadratc decson boundar Lets us model a quadratc decson boundar 3 Decson boundar

7 Lnear dscrmnant analss LDA When covarances are the same ~ N μ, Σ, ~ N μ, Σ, LDA: Lnear decson boundar Contours of class-condtonal denstes

8 LDA: lnear decson boundar Decson boundar Generatve approach to classfcaton Logstc regresson: learns a model of p Generatve approach:. Represents and learns the jont dstrbuton p,. Uses t to defne probablstc dscrmnant functons g o p g p pcal jont model p, p p p = Class-condtonal dstrbutons denstes bnar classfcaton: to class-condtonal dstrbutons p p p = Prors on classes - probablt of class bnar classfcaton: Bernoull dstrbuton p p 8

9 Naïve Baes classfer A generatve classfer model th addtonal smplfng assumptons: All nput attrbutes are condtonall ndependent of each other gven the class. So e have: Y p, p p p d p X X X d Learnng parameters of the model Much smpler denst estmaton problems We need to learn: p and p and p Because of the assumpton of the condtonal ndependence e need to learn: for ever varable : p and p Much easer f the number of nput attrbutes s large Also, the model gves us a fleblt to represent nput attrbutes of dfferent forms!!! E.g. one attrbute can be modeled usng the Bernoull, the other usng Gaussan denst, or as a Posson dstrbuton 9

10 Makng a class decson for the Naïve Baes Dscrmnant functons Posteror of a class choose the class th better posteror probablt p p then = else = g g p d d p, p d p, p p, p Back to logstc regresson o models th lnear decson boundares: Logstc regresson Generatve model th Gaussans th the same covarance matrces ~ N, for ~ N, for o models are related!!! When e have Gaussans th the same covarance matr the probablt of gven has the form of a logstc regresson model!!! p, μ, μ, Σ g

11 When s the logstc regresson model correct? Members of the eponental faml can be often more naturall descrbed as θ A θ f θ, φ h, φ ep a φ θ - A locaton parameter φ - A scale parameter Clam: A logstc regresson s a correct model hen class condtonal denstes are from the same dstrbuton n the eponental faml and have the same scale factor φ Ver poerful result!!!! We can represent posterors of man dstrbutons th the same small netork Lnear regresson Lnear unts f Logstc regresson f f p, g z f p d d d d Gradent update: n f Onlne: Gradent update: f he same f Onlne: n f

12 Gradent-based learnng he same smple gradent update rule derved for both the lnear and logstc regresson models Where the magc comes from? Under the log-lkelhood measure the functon models and the models for the output selecton ft together: Lnear model + Gaussan nose Gaussan nose ~ N, d Logstc + Bernoull Bernoull p g d d z g Bernoull tral d Generalzed lnear models GLIM Assumptons: he condtonal mean epectaton s: f Where f. s a response functon Output s characterzed b an eponental faml dstrbuton th a condtonal mean Eamples: Gaussan nose Lnear model + Gaussan nose ~ N, Logstc + Bernoull Bernoull g e d d d d z g Bernoull tral

13 Generalzed lnear models A canoncal response functons f. : encoded n the dstrbuton θ A θ p θ, φ h, φ ep a φ Leads to a smple gradent form Eample: Bernoull dstrbuton p log ep log log e Logstc functon matches the Bernoull When does the logstc regresson fal? Quadratc decson boundar s needed 3 Decson boundar

14 When does the logstc regresson fal? Another eample of a non-lnear decson boundar Non-lnear etenson of logstc regresson use feature bass functons to model nonlneartes the same trck as used for the lnear regresson Lnear regresson f j m j j j - an arbtrar functon of Logstc regresson f g m j j j d m m 4

15 Evaluaton of classfers Evaluaton For an data set e use to test the classfcaton model on e can buld a confuson matr: Counts of eamples th: class label j that are classfed th a label target 4 7 predct 54 5

16 Evaluaton For an data set e use to test the model e can buld a confuson matr: target 4 7 predct 54 agreement Error:? Evaluaton For an data set e use to test the model e can buld a confuson matr: target predct 4 Error: = 37/3 Accurac = - Error = 94/ agreement 6

17 Evaluaton for bnar classfcaton Entres n the confuson matr for bnar classfcaton have names: target P FP predct FN N P: rue postve ht FP: False postve false alarm N: rue negatve correct rejecton FN: False negatve a mss Addtonal statstcs Senstvt recall SENS P P FN target predct P FN FP N 7

18 Addtonal statstcs Senstvt recall Specfct SENS SPEC P P FN N N FP Postve predctve value precson P PP P FP Negatve predctve value N NPV N FN Bnar classfcaton: addtonal statstcs Confuson matr target predct 4 8 SENS4/6 SPEC8/9 PPV 4/5 NPV 8/ Ro and column quanttes: Senstvt SENS Specfct SPEC Postve predctve value PPV Negatve predctve value NPV 8

19 Classfers Project dataponts to one dmensonal space: Defned for eample b: or p=, Decson boundar Decson boundar = Bnar decsons: Recever Operatng Curves * Probabltes: SENS SPEC p p * * 9

20 Recever Operatng Characterstc ROC ROC curve plots : SN= p * -SP= p * for dfferent * * SENS p * SPEC p * ROC curve Case Case Case p * p *

21 Recever operatng characterstc ROC shos the dscrmnablt beteen the to classes under dfferent decson bases Decson bas can be changed usng dfferent loss functon Qualt of a classfcaton model: Area under the ROC Best value, orst no dscrmnablt:.5