Logistic Regression. Introduction to Data Science Algorithms Jordan Boyd-Graber and Michael Paul SLIDES ADAPTED FROM HINRICH SCHÜTZE

Transcription

1 Logistic Regression Introduction to Data Science Algorithms Jordan Boyd-Graber and Michael Paul SLIDES ADAPTED FROM HINRICH SCHÜTZE Introduction to Data Science Algorithms Boyd-Graber and Paul Logistic Regression of 6

2 What are we talking about? Statistical classification: p(y x) y is typically a Bernoulli or multinomial outcome Classification uses: ad placement, spam detection Building block of other machine learning methods Introduction to Data Science Algorithms Boyd-Graber and Paul Logistic Regression 2 of 6

3 Logistic Regression: Definition Weight vector β i Observations X i Bias β 0 (like intercept in linear regression) P(Y = 0 X) = + exp β 0 + i β () ix i P(Y = X) = exp β 0 + i β ix i + exp β 0 + i β (2) ix i For shorthand, we ll say that P(Y = 0 X) = σ( (β 0 + β i X i )) (3) Where σ(z) = i P(Y = X) = σ( (β 0 + β i X i )) (4) +exp[ z] Introduction to Data Science Algorithms Boyd-Graber and Paul Logistic Regression 3 of 6 i

4 What s this exp doing? Exponential exp[x] is shorthand for e x e is a special number, about Logistic e x is the limit of compound interest formula as compounds become infinitely small It s the function whose derivative is itself The logistic function is σ(z) = +e z Looks like an S Always between 0 and. Introduction to Data Science Algorithms Boyd-Graber and Paul Logistic Regression 4 of 6

5 What s this exp doing? Exponential exp[x] is shorthand for e x e is a special number, about Logistic e x is the limit of compound interest formula as compounds become infinitely small It s the function whose derivative is itself The logistic function is σ(z) = +e z Looks like an S Always between 0 and. Allows us to model probabilities Different from linear regression Introduction to Data Science Algorithms Boyd-Graber and Paul Logistic Regression 4 of 6

6 bias β 0 0. viagra β 2.0 mother β 2.0 Example : Empty Document? X = {} What does Y = mean?

7 bias β 0 0. viagra β 2.0 mother β 2.0 Example : Empty Document? X = {} P(Y = 0) = P(Y = ) = +exp[0.] = exp[0.] +exp[0.] = What does Y = mean?

8 bias β 0 0. viagra β 2.0 mother β 2.0 What does Y = mean? Example : Empty Document? X = {} P(Y = 0) = +exp[0.] = 0.48 P(Y = ) = exp[0.] +exp[0.] = 0.52 Bias β 0 encodes the prior probability of a class

9 bias β 0 0. viagra β 2.0 mother β 2.0 Example 2 X = {Mother, Nigeria} What does Y = mean?

10 bias β 0 0. viagra β 2.0 mother β 2.0 What does Y = mean? Example 2 X = {Mother, Nigeria} P(Y = 0) = P(Y = ) = +exp[ ] = exp[ ] +exp[ ] = Include bias, and sum the other weights

11 bias β 0 0. viagra β 2.0 mother β 2.0 What does Y = mean? Example 2 X = {Mother, Nigeria} P(Y = 0) = 0. P(Y = ) = exp[ ] = exp[ ] +exp[ ] = Include bias, and sum the other weights

12 bias β 0 0. viagra β 2.0 mother β 2.0 Example 3 X = {Mother, Work, Viagra, Mother} What does Y = mean?

13 bias β 0 0. viagra β 2.0 mother β 2.0 What does Y = mean? Example 3 X = {Mother, Work, Viagra, Mother} P(Y = 0) = +exp[ ] = P(Y = ) = exp[ ] +exp[ ] = Multiply feature presence by weight

14 bias β 0 0. viagra β 2.0 mother β 2.0 What does Y = mean? Example 3 X = {Mother, Work, Viagra, Mother} P(Y = 0) = +exp[ ] = 0.60 P(Y = ) = exp[ ] +exp[ ] = 0.30 Multiply feature presence by weight