Machine Learning Linear Models

Transcription

1 Machine Learning Linear Models

2 Outline II - Linear Models 1. Linear Regression (a) Linear regression: History (b) Linear regression with Least Squares (c) Matrix representation and Normal Equation Method (d) Iterative Method: Gradient descent (e) Pros and Cons of both methods 2. Linear Classification: Perceptron (a) Definition and history (b) Example (c) Algorithm

3 Supervised Learning Training data: examples x with labels y. (x 1,y 1 ),...,(x n,y n ) /x i 2 R d Regression: y is a real value, y 2 R f : R d! R f is called a regressor. Classification: y is discrete. To simplify, y 2{ 1, +1} f : R d! { 1, +1} f is called a binary classifier.

4 : History A very popular technique. Rooted in Statistics. Method of Least Squares used as early as 1795 by Gauss. Re-invented in 1805 by Legendre. Frequently applied in astronomy to study the large scale of the universe. Still a very useful tool today. Carl Friedrich Gauss

5 Given: Training data: (x 1,y 1 ),...,(x n,y n ) /x i 2 R d and y i 2 R

6 Given: Training data: (x 1,y 1 ),...,(x n,y n ) /x i 2 R d and y i 2 R example x 1! x 11 x x 1d y 1 label example x i! x i1 x i2... x id y i label example x n! x n1 x n2... x nd y n label

7 Given: Training data: (x 1,y 1 ),...,(x n,y n ) /x i 2 R d and y i 2 R example x 1! x 11 x x 1d y 1 label example x i! x i1 x i2... x id y i label example x n! x n1 x n2... x nd y n label Task: Learn a regression function: f : R d! R f(x) =y

8 Given: Training data: (x 1,y 1 ),...,(x n,y n ) /x i 2 R d and y i 2 R example x 1! x 11 x x 1d y 1 label example x i! x i1 x i2... x id y i label example x n! x n1 x n2... x nd y n label Task: Learn a regression function: f : R d! R f(x) =y Linear Regression: A regression model is said to be linear if it is represented by a linear function.

9 Y #" X 2!" X 1 d = 1, line in R 2 d = 2, hyperplane is R 3 Credit: Introduction to Statistical Learning.

10 Linear Regression Model: f(x) = 0 + dx j=1 jx j with j 2 R, j 2{1,...,d} s are called parameters or coe cients or weights.

11 Linear Regression Model: f(x) = 0 + dx j=1 jx j with j 2 R, j 2{1,...,d} s are called parameters or coe cients or weights. Learning the linear model! learning the 0 s

12 Linear Regression Model: f(x) = 0 + dx j=1 jx j with j 2 R, j 2{1,...,d} s are called parameters or coe cients or weights. Learning the linear model! learning the 0 s Estimation with Least squares: Use least square loss: `oss(y i,f(x i )) = (y i f(x i )) 2

13 Linear Regression Model: f(x) = 0 + dx j=1 jx j with j 2 R, j 2{1,...,d} s are called parameters or coe cients or weights. Learning the linear model! learning the 0 s Estimation with Least squares: Use least square loss: `oss(y i,f(x i )) = (y i f(x i )) 2 We want to minimize the loss over all examples, that is minimize the risk or cost function R: R = 1 2n (y i f(x i )) 2

14 Asimplecasewithonefeature(d =1): f(x) = x

15 Asimplecasewithonefeature(d =1): We want to minimize: f(x) = x R = 1 2n (y i f(x i )) 2

16 Asimplecasewithonefeature(d =1): We want to minimize: f(x) = x R = 1 2n R( )= 1 2n (y i f(x i )) 2 (y i 0 1 x i ) 2

17 Asimplecasewithonefeature(d =1): We want to minimize: f(x) = x R = 1 2n R( )= 1 2n Find 0 and 1 that minimize: R( )= 1 2n (y i f(x i )) 2 (y i 0 1 x i ) 2 (y i 0 1 x i ) 2

18 β RSS β 0 β β 0 Credit: Introduction to Statistical Learning.

19 Find 0 and 1 so that: argmin ( 1 2n (y i 0 1 x i ) 2 )

20 Find 0 and 1 so that: Minimize: R( 0, argmin ( 1 2n 1), that is: (y i 0 1 x i ) =0

21 Find 0 and 1 so that: Minimize: R( 0, =2 0 2n argmin ( 1 2n 1), that is: (y i 0 1 x i ) =0 (y i 0 1 x i (y i 0 1 x i 0

22 Find 0 and 1 so that: Minimize: R( 0, =2 0 2n argmin ( 1 2n 1), that is: (y i 0 1 x i ) =0 (y i 0 1 x i (y i 0 1 x i 0 = 1 n (y i 0 1 x i ) ( 1) = 0

23 Find 0 and 1 so that: Minimize: R( 0, =2 0 2n argmin ( 1 2n 1), that is: (y i 0 1 x i ) =0 (y i 0 1 x i (y i 0 1 x i 0 = 1 n (y i 0 1 x i ) ( 1) = 0 0 = 1 n y i 1 1 n x i

24 =2 1 2n (y i 0 1 x i (y i 0 1 x i 1

25 =2 1 2n (y i 0 1 x i (y i 0 1 x i 1 = 1 n (y i 0 1 x i ) ( x i )=0

26 =2 1 2n (y i 0 1 x i (y i 0 1 x i 1 = 1 n (y i 0 1 x i ) ( x i )=0 1 x i 2 = x i y i n X 0x i

27 =2 1 2n (y i 0 1 x i (y i 0 1 x i 1 = 1 n (y i 0 1 x i ) ( x i )=0 Plugging 0 in 1 : 1 x i 2 = x i y i n X 0x i 1 = P n y i x i P n x 2 i 1 n P n y i P n x i 1 n P n x i P xi

28 With more than one feature: Find the f(x) = 0 + j that minimize: dx j=1 jx j R = 1 2n (y i 0 d X j=1 jx ij )) 2 Let s write it more elegantly with matrices!

29 Matrix representation Let X be an n (d + 1) matrix where each row starts with a 1 followed by a feature vector. Let y be the label vector of the training set. Let be the vector of weights (that we want to estimate!). X := 0 B B B B B 1 x 11 x 1j x 1d x i1 x ij x id x n1 x nj x nd 1 C C C C C C A y := 0 B B B B B y 1. y i. y n 1 C C C C C C A := 0 B B B B B 0. j. d 1 C C C C C C A

30 Normal Equation We want to find (d + 1) s that minimize R. We write R: We have that: R( )= 1 (y X ) 2 2n R( )= 1 2n (y X )T (y X = 1 n XT (y X 2 = 1 n XT X is positive definite which ensures that X T (y X )=0 is a minimum. We solve: The unique solution is: =(X T X) 1 X T y

31 Gradient descent!"#$%&'() ' 4.5' +,-./0$'12,3$' *$%&'( '

32 Gradient descent Gradient Descent is an optimization method. Repeat until convergence: Update simultaneously all j for (j = 0 and j = 1) 0 0 R( 0, 1) 1 1 R( 0, 1)

33 Gradient descent Gradient Descent is an optimization method. Repeat until convergence: Update simultaneously all j for (j = 0 and j = 1) 0 0 R( 0, 1) 1 1 R( 0, 1) is a learning rate.

34 Gradient descent In the linear 0 = 1 n (y i 0 1 x i ) ( 1) = 0 Let s generalize 1 = 1 n (y i 0 1 x i ) ( x i )

35 Gradient descent In the linear 0 = 1 n (y i 0 1 x i ) ( 1) = 1 = 1 n (y i 0 1 x i ) ( x i ) Repeat until convergence: Update simultaneously all j for (j = 0 and j = 1) 0 := 0 1 n ( x i y i ) 1 := 1 1 n ( x i y i )(x i )

36 Pros and Cons Analytical approach: Normal Equation + No need to specify a convergence rate or iterate. - Works only if X T X is invertible - Very slow if d is large O(d 3 ) to compute (X T X) 1 Iterative approach: Gradient Descent + E ective and e cient even in high dimensions. - Iterative (sometimes need many iterations to converge). - Needs to choose the rate.

37 Practical considerations 1. Scaling: Bring your features to a similar scale.

38 Practical considerations 1. Scaling: Bring your features to a similar scale. x i := x i µ i stdev(x i )

39 Practical considerations 1. Scaling: Bring your features to a similar scale. x i := x i µ i stdev(x i ) 2. Learning rate: Don t use a rate that is too small or too large.

40 Practical considerations 1. Scaling: Bring your features to a similar scale. x i := x i µ i stdev(x i ) 2. Learning rate: Don t use a rate that is too small or too large. 3. Rshoulddecreaseafter each iteration.

41 Practical considerations 1. Scaling: Bring your features to a similar scale. x i := x i µ i stdev(x i ) 2. Learning rate: Don t use a rate that is too small or too large. 3. Rshoulddecreaseafter each iteration. 4. Declare convergence if it start decreasing by less

42 Practical considerations 1. Scaling: Bring your features to a similar scale. x i := x i µ i stdev(x i ) 2. Learning rate: Don t use a rate that is too small or too large. 3. Rshoulddecreaseafter each iteration. 4. Declare convergence if it start decreasing by less 5. If X T X is not invertible?

43 Practical considerations 1. Scaling: Bring your features to a similar scale. x i := x i µ i stdev(x i ) 2. Learning rate: Don t use a rate that is too small or too large. 3. Rshoulddecreaseafter each iteration. 4. Declare convergence if it start decreasing by less 5. If X T X is not invertible? (a) Too many features as compared to the number of examples (e.g., 50 examples and 500 features) (b) Features linearly dependent: e.g., weight in pounds and in kilo.

44 Credit The elements of statistical learning. Data mining, inference, and prediction. 10th Edition T. Hastie, R. Tibshirani, J. Friedman. Machine Learning Tom Mitchell.