CS 340 Lec. 16: Logistic Regression

Transcription

1 CS 34 Lec. 6: Logistic Regression AD March AD ) March / 6

2 Introduction Assume you are given some training data { x i, y i } i= where xi R d and y i can take C different values. Given an input test data x, you want to predict/estimate the output y associated to x. Previously we have followed a probabilistic approach p y = k x) = p x y = k) p y = k) C j= p x y = j) p y = j). This requires modelling and learning the parameters of the class conditional density of features p x y = k). AD ) March / 6

3 Logistic Regression Discriminative model: we model and learn directly p y = k x) and bypassing the introduction of p x y = k). Consider the following model for C = binary classification) p y = x, w) = p y = x, w) ) = g w T x where w = w w d ) T, x = x x d ) T so z = w T x =w + d j= w j x j and g is a squashing function: g : R [, ]. Logistic regression corresponds to g z) = exp z) = + exp z) + exp z). AD ) March 3 / 6

4 Logistic Function introb a) Left) logistic or sigmoid function Right) logistic regression for x=sat.5: a) The sigmoid or logistic function. Produced by sigmoidplot. b) Logistic regression for SAT scores. Solid black score and y=pass/fail class solid black dots are the data), open red ta. The open red circles is the predicted probability. The green crosses denote two students with the same SAT score of 55 an input circles representation are predicted x) but with different probabilities. training labels one student passed, y =, the other failed, y = ). Hence this dat tly separable using just the SAT feature. Figure generated by logregsatdemo. AD ) March 4 / 6 b)

5 Logistic Regression The log odds ratio satisfies LOR x) = log p y = x, w) p y = x, w) = wt x so the logistic parameters are easily interpretable. If w j >, then increasing x j makes y = more likely while decreasing x j makes y = more likely and opposite if w j = ). w j = means x j has no impact on the outcome. Logistic regression partitions the input space into two regions whose decision boundary is {x :LOR x) = } = { x : w T x = } Simple model of a neuron: it forms a weighted sum of its inputs and the fires an output pulse if this sum exceeds a threshold. Logistic regression mimics this as you can sort of think of it as a process which fires if p y = x, w) > p y = x, w) equivalently if LOR x) >. AD ) March 5 / 6

6 Logistic Function in Two Dimensions Plots of p y = w x + w x ). Here w = w, w ) define the normal to the decision boundary. Points to the right have w T x > and to the left have w T x <. + w x ). Here w = w, w ) defines the normal to the decision bounda points toad the) left have sigmw T x) <.5. Based on FigureMarch 39.3 of 6 [Mac / 6

7 MLE Parameter Learning for Logistic Regression To learn the parameters w, we can maximize w.r.t w the conditional) log-likelihood function {y i l w) = log p } We have l w) = = i= = i= log p y i x i, w ) i= i= { x i } ) i=, w = log i= p y i x i, w ) y i log p y i = x i, w ) + y i ) log p y i = x i, w ) y i ) w T x i i= log + exp w )) T x i Good news: l w) is concave so there is no local maxima. Bad news: there is no-closed form solution for ŵ MLE. AD ) March 7 / 6

8 Gradient Ascent Gradient ascent is one of the most basic method to maximize a function. It is an iterative procedure such that at iteration t : where the gradient is w t) = w t ) + η w l w) w t ) w l w) = [ lw) w lw) w d ] T and η > is the learning rate. To minimize a function f w), simply use the gradient descent w t) = w t ) η w f w) w t ) AD ) March 8 / 6

9 Gradient Descent Example compb Gradient descent a) on a simple function, starting from,) b) for steps using η =. left) and η =.6 right).: Gradient descent on a simple function, starting from, ), for steps, using a fixed learning rate step size) η. Th is at, ). a) η =.. b) η =.6. Produced by steepestdescentdemo. AD ) March 9 / 6

10 Gradient Ascent for Logistic Regression We have l w) w k Hence we have = l w) w k = = i= y i ) x i k + xk i i= xk i i= So in vector-form, we will do i= x i k exp w T x i ) + exp w T x i ) { p y i = x i, w ) y i )} { y i p y i = x i, w )} w t) = w t ) + η w l w) w t ) { = w t ) + η y i p i= y i = x i, w t ))} x i AD ) March / 6

11 Regularized Logistic Regression Similarly to regression, we can regularize the solution by assigning a Gaussian prior to w p w) = d j= p w j ) = d j= w j ;, λ) This pushes the parameters w towards zero and can prevent overfitting. In this case, we have ŵ MAP = arg max p w { x i, y i } ) i= = arg max l w) wt w λ. ŵ MAP can be computed iteratively using ) w t) = w t ) + η w l w) wt w λ { { wt ) = w t ) + η λ w+ i= y i p y i = x i, w t ))} } x i AD ) March / 6

12 Using Basis Functions for Logistic Regression Similarly to regression, we can use basis functions; i.e. ) p y = x, w) = g w T Φ x) where w = w w m ) T, Φ x) = Φ x) Φ m x)) T. For example, if x R then we can pick Φ x) =, x,..., x m ) For x R d, we can pick some radial basis functions ) T ) x µ j x µ j Φ j x) = exp σ. AD ) March / 6

13 Example a) b) left) Logistic regression in the original feature space x = x, x ). right) Logistic after space. performing nd degree poly degree expansion gistic regresionregression model in theobtained original feature b) Aftera performing a second polynomi Φ x =, x, x, x, x. ) egbasisfndemo. AD ) March 3 / 6

14 l inexample the original feature space. b) After performing a second degree polynomial expansion. F poly rbf prototypes b) c) left) Logistic regression for Φ x) =, x, x,..., x, x. right) Logistic regression using 4 radial basis functions with centers µ speci ed g a linear logistic regression classifier using degree polynomial expansion.j c) Same mod by black crosses. fied by the 4 black crosses. Figure generated by logregxordemo. AD ) March 4 / 6

15 Multinomial Logistic Regression Consider now the case where C >. We could consider the following generalization ) p y = c x, {w c } C c= = exp wc T x ) C k= exp for c =,..., C wk T x) but this is not identifiable as ) p y = c x, {w c + w } C c= ) = p y = c x, {w c } C c=. Hence we set w C = ) T to obtain ) p y = c x, {w c } C exp wc T x ) c= = + C k= exp wk T p ) y = C x, {w c } C c= = + C k= exp w T k x). x) for c =,..., C The conditional) log-likelihood is concave w.r.t {w c } C c= so MLE estimates can be computed using gradient. AD ) March 5 / 6

16 Figure.9: Softmax distribution Sη/T ), where η = 3,, ), at different temperatures T. When the temperature is high left), the Example distribution is uniform, whereas when the temperature is low right), the distribution is spiky, with all its mass on the largest element. Figure generated by softmaxdemo. Linear Multinomial Logistic Regression Kernel RBF Multinomial Logistic Regression a) b) c) left) Some 5 class data in d center) Multinomial logistic regression in the original feature space x = x Figure.: a) Some 5 class data in d. b) Multinomial, x logistic ) right) RBF basis functions with regression in the original feature space. c) RBF basis functions with bandwidth using m = +. We use all the data points as centers. bandwidth of. We use all the data points as centers. Figure generated by logregmultinomkerneldemo. AD ) March 6 / 6