Machine Learning - Waseda University Logistic Regression

Transcription

1 Machine Learning - Waseda University Logistic Regression AD June AD ) June / 9

2 Introduction Assume you are given some training data { x i, y i } i= where xi R d and y i can take C different values. Given an input test data x, you want to predict/estimate the output y associated to x. A common approach consists of using p y = k x) = p x y = k) p y = k) C j= p x y = j) p y = j). This requires modelling and learning the parameters of the class conditional density of features p x y = k). This can be tedious for complicated problems. AD ) June / 9

3 Logistic Regression Discriminative model: we model and learn directly p y = k x) and bypassing the introduction of p x y = k). Consider the following model for C = binary classification) p y = x, w) = p y = x, w) ) = g w T x where w = w w d ) T, x = x x d ) T so z = w T x =w + d j= w j x j and g is a squashing function: g : R [, ]. Logistic regression corresponds to g z) = exp z) = + exp z) + exp z). AD ) June 3 / 9

4 Logistic Function introb a) Left) logistic or sigmoid function Right) logistic regression for x=sat.5: a) The sigmoid or logistic function. Produced by sigmoidplot. b) Logistic regression for SAT scores. Solid black score and y=pass/fail class solid black dots are the data), open red ta. The open red circles is the predicted probability. The green crosses denote two students with the same SAT score of 55 an input circles representation are predicted x) but with different probabilities. training labels one student passed, y =, the other failed, y = ). Hence this dat tly separable using just the SAT feature. Figure generated by logregsatdemo. AD ) June 4 / 9 b)

5 Logistic Regression The log odds ratio satisfies LOR x) = log p y = x, w) p y = x, w) = wt x so the logistic parameters are easily interpretable. If w j >, then increasing x j makes y = more likely while decreasing x j makes y = more likely and opposite if w j = ). w j = means x j has no impact on the outcome. Logistic regression partitions the input space into two regions whose decision boundary is {x :LOR x) = } = { x : w T x = } Simple model of a neuron: it forms a weighted sum of its inputs and the fires an output pulse if this sum exceeds a threshold. Logistic regression mimics this as you can sort of think of it as a process which fires if p y = x, w) > p y = x, w) equivalently if LOR x) >. AD ) June 5 / 9

6 Logistic Function in Two Dimensions Plots of p y = w x + w x ). Here w = w, w ) define the normal to the decision boundary. Points to the right have w T x > and to the left have w T x <. + w x ). Here w = w, w ) defines the normal to the decision bounda points toad the) left have sigmw T x) <.5. Based on FigureJune 39.3 of 6 [Mac / 9

7 Using Basis Functions for Logistic Regression Similarly to regression, we can use basis functions; i.e. ) p y = x, w) = g w T Φ x) where w = w w m ) T, Φ x) = Φ x) Φ m x)) T. For example, if x R then we can pick Φ x) =, x,..., x m ) For x R d, we can pick some radial basis functions ) T ) x µ j x µ j Φ j x) = exp σ. AD ) June 7 / 9

8 Example a) b) left) Logistic regression in the original feature space x = x, x ). right) Logistic after space. performing nd degree poly degree expansion gistic regresionregression model in theobtained original feature b) Aftera performing a second polynomi Φ x =, x, x, x, x. ) egbasisfndemo. AD ) June 8 / 9

9 l inexample the original feature space. b) After performing a second degree polynomial expansion. F poly rbf prototypes b) c) left) Logistic regression for Φ x) =, x, x,..., x, x. right) Logistic regression using 4 radial basis functions with centers µ speci ed g a linear logistic regression classifier using degree polynomial expansion.j c) Same mod by black crosses. fied by the 4 black crosses. Figure generated by logregxordemo. AD ) June 9 / 9

10 MLE Parameter Learning for Logistic Regression To learn the parameters w, we can maximize w.r.t w the conditional) log-likelihood function {y i l w) = log p } We have l w) = = i= = i= log p y i x i, w ) i= i= { x i } ) i=, w = log i= p y i x i, w ) y i log p y i = x i, w ) + y i ) log p y i = x i, w ) y i ) w T Φ x i ) i= log + exp w T Φ x i ))) Good news: l w) is concave so there is no local maxima. Bad news: there is no-closed form solution for ŵ MLE. AD ) June / 9

11 Gradient Ascent Gradient ascent is one of the most basic method to maximize a function. It is an iterative procedure such that at iteration t : where the gradient is w t) = w t ) + η w l w) w t ) w l w) = and η > is the learning rate. [ lw) w lw) w m ] T AD ) June / 9

12 Gradient Descent Example compb Gradient descent a) on a simple function, starting from,) b) for steps using η =. left) and η =.6 right).: Gradient descent on a simple function, starting from, ), for steps, using a fixed learning rate step size) η. Th is at, ). a) η =.. b) η =.6. Produced by steepestdescentdemo. AD ) June / 9

13 Gradient Ascent for Logistic Regression We have l w) = where [Φ] i,j = Φ j x i ), y = y y i g w T Φ x i ))) Φ x i ) = Φ T y µ) i= y ) T and µ = g w T Φ x )) g w T Φ x ))) T. So in vector-form, we will do w t) = w t ) + η w l w) w t ) = w t ) + η Φ T y µ t )) where µ t ) corresponds to µ computed using w t ). AD ) June 3 / 9

14 Iterative Reweighted Least Squares ewton s method is a generic second order) optimization algorithm which converges faster than the simple gradient algorithm. It proceeds as follows at iteration t We have w t) = w t ) [ l w t ))] l w t )). l w) = Φ T UΦ with U a diagonal matrix with diagonal element [U] i,i = g w T Φ x i )) [ g w T Φ x i ))]. It can be written as w t) = Φ T U Φ) t ) Φ T U t ) Φw t ) + [U t )] y µ t ))) where U t ) and µ t ) corresponds to U and µ with w t ). AD ) June 4 / 9

15 Regularized Logistic Regression via Gaussian Prior Similarly to regression, we can regularize the solution by assigning a Gaussian prior to w p w) = m m p w k ) = w k ;, λ) k= k= This pushes the parameters w towards zero and can prevent overfitting. In this case, we have w MAP = arg max p w { x i, y i } ) i= = arg max l w) wt w λ. w MAP can be computed iteratively using w t ) + η { λ w t ) + Φ T y µ t ))} Regularization parameter can be estimated using cross-validation or by maximizing marginal likelihood. AD ) June 5 / 9

16 Regularized Logistic Regression via Laplace Prior Similarly to regression, we can regularize the solution by assigning a Gaussian prior to w m m p w) = p w k ) = λ exp w ) k λ k= k= This pushes the parameters w towards zero and can prevent overfitting. In this case, we have w MAP = arg max p w { x i, y i } ) i= = arg max l w) m λ w k. k= The objective function is convex and effi cient procedures have been developed to compute w MAP. Similarly to the regression case, this can lead to sparse solution; e.g. you can have w k,map = exactly. Regularization parameter can be estimated using cross-validation or by maximizing marginal likelihood. AD ) June 6 / 9

17 Multinomial Logistic Regression Consider now the case where C >. We could consider the following generalization ) p y = c x, {w c } C c= = exp wc T Φ x) ) C k= exp for c =,..., C wk T Φ x)) but this is not identifiable as ) p y = c x, {w c + w } C c= Hence we set w C = ) T to obtain ) p y = c x, {w c } C c= ) p y = C x, {w c } C c= = = ) = p y = c x, {w c } C c=. exp w T c Φ x) ) + C k= exp for c =,..., C wk T Φ x)) + C k= exp wk TΦ x)). The conditional) log-likelihood is concave w.r.t {w c } C c= so MLE estimates can be computed using gradient. AD ) June 7 / 9

18 Figure.9: Softmax distribution Sη/T ), where η = 3,, ), at different temperatures T. When the temperature is high left), the Example distribution is uniform, whereas when the temperature is low right), the distribution is spiky, with all its mass on the largest element. Figure generated by softmaxdemo. Linear Multinomial Logistic Regression Kernel RBF Multinomial Logistic Regression a) b) c) left) Some 5 class data in d center) Multinomial logistic regression in the original feature space x = x Figure.: a) Some 5 class data in d. b) Multinomial, x logistic ) right) RBF basis functions with regression in the original feature space. c) RBF basis functions with bandwidth using m = +. We use all the data points as centers. bandwidth of. We use all the data points as centers. Figure generated by logregmultinomkerneldemo. AD ) June 8 / 9

19 Full Bayesian Analysis of Logistic Regression Even for Gaussian priors on w, one cannot compute {y {y i p } i= { x i } ) i=, λ = p i } i= { x i } ) i=, w p w λ) ) p {y i } i= {x i } i=, λ where {y i p } i= { x i } ) i=, λ = {y i p } i= { x i } ) i=, w p w λ) dw Contrary to regression, there is no closed form Bayesian analysis possible. If you want to do Bayesian inference, then approximations are necessary. AD ) June 9 / 9