Logistic Regression. and Maximum Likelihood. Marek Petrik. Feb

Transcription

1 Lgistic Regressin and Maximum Likelihd Marek Petrik Feb

2 S Far in ML Regressin vs Classificatin Linear regressin Bias-variance decmpsitin Practical methds fr linear regressin

3 Simple Linear Regressin We have nly ne feature Y β 0 + β 1 X Y = β 0 + β 1 X + ɛ Example: TV Sales sales β 0 + β 1 TV

4 Multiple Linear Regressin Y X 2 X 1

5 Types f Functin f Regressin: cntinuus target f : X R Years f Educatin Senirity Incme Classificatin: discrete target f : X {1, 2, 3,..., k} X1 X2

6 Tday Why nt use linear regressin fr classificatin Lgistic regressin Maximum likelihd principle Maximum likelihd fr linear regressin Reading: ISL ESL 2.6 (max likelihd)

7 Examples f Classificatin 1. A persn arrives at the emergency rm with a set f symptms that culd pssibly be attributed t ne f three medical cnditins. Which f the three cnditins des the individual have?

8 Examples f Classificatin 2. An nline banking service must be able t determine whether r nt a transactin being perfrmed n the site is fraudulent, n the basis f the userffs IP address, past transactin histry, and s frth.

9 Examples f Classificatin 3. On the basis f DNA sequence data fr a number f patients with and withut a given disease, a bilgist wuld like t figure ut which DNA mutatins are deleterius (disease-causing) and which are nt.

10 IBM Watsn Fair use, Lgistic regressin + clever functin engineering

11 Predicting Default default f(incme, balance) Incme Balance

12 Predicting Default default f(incme, balance) Bxplt Balance Incme N Yes Default N Yes Default

13 Casting Classificatin as Regressin Regressin: f : X R Classificatin: f : X {1, 2, 3}

14 Casting Classificatin as Regressin Regressin: f : X R Classificatin: f : X {1, 2, 3} But {1, 2, 3} R D we even need classificatin?

15 Casting Classificatin as Regressin Regressin: f : X R Classificatin: f : X {1, 2, 3} But {1, 2, 3} R D we even need classificatin? Yes! Regressin: Values that are clse are similar Classificatin: Distance f classes is meaningless

16 Casting Classificatin as Regressin: Example Predict pssible diagnsis: {strke, verdse, seizure} Assign class labels: 1 if strke Y = 2 if verdse 3 if seizure. Fit linear regressin

17 Casting Classificatin as Regressin: Example Predict pssible diagnsis: {strke, verdse, seizure} Assign class labels: 1 if strke Y = 2 if verdse 3 if seizure. Fit linear regressin Make predictins: If uncertain whether symptms pint t strke r seizure, we predict verdse

18 Linear Regressin fr 2-class Classificatin Y = { 1 if default 0 therwise Linear regressin Lgistic regressin Balance Prbability f Default Balance Prbability f Default P[default = yes balance]

19 Lgistic Regressin Predict prbability f a class: p(x) Example: p(balance) prbability f default fr persn with balance Linear regressin: lgistic regressin: p(x) = β 0 + β 1 p(x) = eβ 0+β 1 X 1 + e β 0+β 1 X the same as: ( ) p(x) lg = β 0 + β 1 X 1 p(x) Odds: p(x) /1 p(x)

20 Lgistic Functin y = ex 1 + e x Lgistic x

21 Lgistic Functin ( ) p(x) lg 1 p(x) Lgit p(x)

22 Lgistic Regressin P[default = yes balance] = eβ 0+β 1 balance 1 + e β 0+β 1 balance Linear regressin Lgistic regressin Balance Prbability f Default Balance Prbability f Default

23 Estimating Cefficients: Maximum Likelihd Likelihd: Prbability that data is generated frm a mdel Find the mst likely mdel: l(mdel) = P[data mdel] max l(mdel) = max P[data mdel] mdel mdel Likelihd functin is difficult t maximize Transfrm it using lg (strictly increasing) max lg l(mdel) mdel Strictly increasing transfrmatin des nt change maximum

24 Example: Maximum Likelihd Assume a cin with p as the prbability f heads Data: h heads, t tails The likelihd functin is: l(p) = p h (1 p) t. Likelihd 0e+00 2e 07 4e 07 6e 07 8e p

25 Likelihd Functin: 2 cin flips heads h = 1 tails t = 1 Likelihd p

26 Likelihd Functin: 20 cin flips heads h = 10 tails t = 10 Likelihd 0e+00 2e 07 4e 07 6e 07 8e p

27 Likelihd Functin: 200 cin flips heads h = 100 tails t = 100 Likelihd 0e+00 2e 61 4e 61 6e p

28 p Maximizing Likelihd Likelihd functin is nt cncave: hard t maximize l(p) = p h (1 p) t. Maximize the lg-likelihd instead lg l(p) = h lg(p) + t lg(1 p). Lglikelihd

29 Lg-likelihd: Biased Cin heads h = 20 tails t = 50 Lglikelihd p

30 Maximize Lg-likelihd Lg-likelihd: lg l(p) = h lg(p) + t lg(1 p).

31 Maximize Lg-likelihd Lg-likelihd: lg l(p) = h lg(p) + t lg(1 p). Maximum where derivative = 0 Derivative: d dp h lg(p) + t lg(1 p) = h p t 1 p

32 Maximize Lg-likelihd Lg-likelihd: lg l(p) = h lg(p) + t lg(1 p). Maximum where derivative = 0 Derivative: d dp h lg(p) + t lg(1 p) = h p t 1 p Maximum likelihd slutin: p = h h + 1

33 Max-likelihd: Lgistic Regressin Features x i and labels y i Likelihd: l(β 0, β 1 ) = p(x i ) (1 p(x i )) i:y i =1 i:y i =0 Lg-likelihd: l(β 0, β 1 ) = lg p(x i ) + lg(1 p(x i )) i:y i =1 i:y i =0 Cncave maximizatin prblem Can be slved using gradient descent

34 Multiple Lgistic Regressin Multiple features eβ 0+β 1 X 1 +β 2 X β mx n p(x) = 1 + e β 0+β 1 X 1 +β 2 X β mx n Equivalent t: ( ) p(x) lg = β 0 + β 1 X 1 + β 2 X β m X n 1 p(x)

35 Multinmial Lgistic Regressin Predicting multiple classes: Medical diagnsis 1 if strke Y = 2 if verdse 3 if seizure. Predicting which prducts custmer purchases Straightfrward generalizatin f simple lgistic regressin e c e c 1 e c 1 e c 1 + e c e c k