Data Mining Techniques. Lecture 2: Regression

Transcription

1 Data Mining Techniques CS Section 3 - Fall 2016 Lecture 2: Regression Jan-Willem van de Meent (credit: Yijun Zhao, Marc Toussaint, Bishop)

2 Administrativa Instructor Jan-Willem van de Meent Phone: Office Hours: 478 WVH, Wed 1.30pm pm Teaching Assistants Yuan Zhong yzhong@ccs.neu.edu Office Hours: WVH 462, Wed 3pm - 5pm Kamlendra Kumar kumark@zimbra.ccs.neu.edu Office Hours: WVH 462, Fri 3pm - 5pm

3 Administrativa Course Website Piazza Project Guidelines (Vote next week)

4 Question What would you like to get out of this course?

5 Linear Regression

6 Regression Examples Features Continuous Value x =) y {age, major, gender, race} GPA {income, credit score, profession} Loan Amount {college,major,gpa} Future Income

7 Example: Boston Housing Data UC Irvine Machine Learning Repository (good source for project datasets)

8 Example: Boston Housing Data 1. CRIM: per capita crime rate by town 2. ZN: proportion of residential land zoned for lots over 25,000 sq.ft. 3. INDUS: proportion of non-retail business acres per town 4. CHAS: Charles River dummy variable (= 1 if tract bounds river; 0 otherwise) 5. NOX: nitric oxides concentration (parts per 10 million) 6. RM: average number of rooms per dwelling 7. AGE: proportion of owner-occupied units built prior to DIS: weighted distances to five Boston employment centres 9. RAD: index of accessibility to radial highways 10. TAX: full-value property-tax rate per $10, PTRATIO: pupil-teacher ratio by town 12. B: 1000(Bk )^2 where Bk is the proportion of african americans by town 13. LSTAT: % lower status of the population 14. MEDV: Median value of owner-occupied homes in $1000's

9 Example: Boston Housing Data CRIM: per capita crime rate by town

10 Example: Boston Housing Data CHAS: Charles River dummy variable (= 1 if tract bounds river; 0 otherwise)

11 Example: Boston Housing Data MEDV: Median value of owner-occupied homes in $1000's

12 Example: Boston Housing Data N data points D features

13 Regression: Problem Setup Given N observations {(x 1, y 1 ), (x 2, y 2 ),...,(x N, y N )} learn a function y i = f (x i ) i = 1, 2,..., N and for a new input x* predict y = f (x )

14 Linear Regression Assume f is a linear combination of D features were x and w are defined as for N points we write Learning task: Estimate w

15 Linear Regression

16 Error Measure Mean Squared Error (MSE): E(w) = 1 N NX n=1 (w T x n y n ) 2 where 2 X = 6 4 x 1 T x 2 T... x N T = 1 N 3 k Xw y k2 7 5 y = y 1 T y 2 T... y N T 3 7 5

17 Minimizing the Error E(w) = 1 N k Xw y k2 5E(w) = 2 N XT (Xw y) =0 X T Xw = X T y w = X y where X =(X T X) 1 X T is the pseudo-inverse of X

18 Minimizing the Error E(w) = 1 N k Xw y k2 5E(w) = 2 N XT (Xw y) =0 X T Xw = X T y w = X y where X =(X T X) 1 X T is the pseudo-inverse of X Matrix Cookbook (on course website)

19 Ordinary Least Squares Construct matrix X and the vector y from the dataset {(x 1, y 1 ), x 2, y 2 ),...,(x N, y N )} (each x includes x 0 =1)asfollows: 2 x T y X = 6 x T 1 T y = 6 y2 T x T N y T N Compute X =(X T X) 1 X T Return w = X y

20 Gradient Descent 50 countours : E(w ) w w 0

21 Least Mean Squares (a.k.a. gradient descent) Initialize the w(0) for time t =0 for t =0, 1, 2,... do Compute the gradient g t = 5E(w(t)) Set the direction to move, v t = g t Update w(t +1)=w(t)+ v t Iterate until it is time to stop Return the final weights w

22 Question When would you want to use OLS, when LMS?

23 Computational Complexity Ordinary least squares (OMS) Least Mean Squares (LMS)

24 Computational Complexity Ordinary least squares (OMS) Least Mean Squares (LMS) OMS is expensive when D is large

25 Effect of step size

26 Choosing Stepsize r rf(x)? Set step size proportional to? small gradient small step? large gradient large step?

27 Choosing Stepsize r rf(x)? Set step size proportional to? small gradient small step? large gradient large step? Two commonly used techniques 1. Stepsize adaptation 2. Line search

28 Stepsize Adaptation Input: initial x 2 R n, functions f(x) and rf(x), initial stepsize, tolerance Output: x 1: repeat 2: y x rf(x) rf(x) 3: if [ thenstep is accepted]f(y) apple f(x) 4: x y 5: 1.2 // increase stepsize 6: else[step is rejected] 7: 0.5 // decrease stepsize 8: end if 9: until y x < [perhaps for 10 iterations in sequence] ( magic numbers )

29 Second Order Methods Compute Hessian matrix of second derivatives

30 Second Order Methods Broyden-Fletcher-Goldfarb-Shanno (BFGS) method: Input: initial x 2 R n, functions f(x), rf(x), tolerance Output: x 1: initialize H -1 = I n 2: repeat 3: compute = H -1 rf(x) 4: perform a line search min f(x + ) 5: 6: y rf(x + ) rf(x) 7: x x + 8: update H -1 9: until 1 < I y > >H -1 I > y y > + > > y > y Memory-limited version: L-BFGS

31 Stochastic Gradient Descent What if N is really large? Batch gradient descent (evaluates all data) Minibatch gradient descent (evaluates subset) Converges under Robbins-Monro conditions

32 Probabilistic Interpretation

33 Normal Distribution Right Skewed Left Skewed Random

34 Normal Distribution ) p Density:

35 Central Limit Theorem 3 N =1 3 N =2 3 N = If y1,, yn are 1. Independent identically distributed (i.i.d.) 2. Have finite variance 0 < σy 2 <

36 Density: Multivariate Normal

37 Regression: Probabilistic Interpretation

38 Regression: Probabilistic Interpretation

39 Regression: Probabilistic Interpretation Joint probability of N independent data points

40 Regression: Probabilistic Interpretation Log joint probability of N independent data points

41 Regression: Probabilistic Interpretation Log joint probability of N independent data points

42 Regression: Probabilistic Interpretation Log joint probability of N independent data points

43 Regression: Probabilistic Interpretation Log joint probability of N independent data points Maximum Likelihood

44 Basis function regression Linear regression y = w 0 + w 1 x w D x D = w T x Basis function regression Polynomial regression

45 Polynomial Regression 1 M =0 1 M =1 t t x 1 0 x 1 1 M =3 1 M =9 t t x 1 0 x 1

46 Polynomial Regression 1 M =0 1 M =1 Underfit t t x 1 0 x 1 1 M =3 1 M =9 t t x 1 0 x 1

47 Polynomial Regression 1 M =0 1 M =1 t t x 1 0 x 1 Overfit 1 M =3 1 M =9 t t x 1 0 x 1

48 Regularization L2 regularization(ridgeregression)minimizes: E(w) = 1 N k Xw y k2 + k w k 2 where 0 and w 2 = w T w k k L1 regularization (LASSO) minimizes: E(w) = 1 N k Xw y k2 + w 1 where 0 and w 1 = D P i=1! i

49 Regularization

50 Regularization L2: closed form solution w =(X T X + I) 1 X T y L1: No closed form solution. Use quadratic programming: minimize k Xw y k 2 s.t. k w k 1 apple s

51 Review: Bias-Variance Trade-off Maximum likelihood estimator Bias-variance decomposition (expected value over possible data points)

52 Bias-Variance Trade-off Often: low bias ) high variance low variance ) high bias Trade-o :

53 K-fold Cross-Validation 1. Divide dataset into K folds 2. Train on all except k-th fold 3. Test on k-th fold 4. Minimize test error w.r.t. λ

54 K-fold Cross-Validation Choices for K: 5, 10, N (leave-one-out) Cost of computation: K x number of λ

55 Learning Curve

56 Learning Curve

57 Loss Functions