Least Mean Squares Regression - PDF Free Download

Least Mean Squares Regression Machine Learning Spring 2018 The slides are mainly from Vivek Srikumar 1

Lecture Overview Linear classifiers What functions do linear classifiers express? Least Squares Method for Regression 2

Where are we? Linear classifiers What functions do linear classifiers express? Least Squares Method for regression Examples The LMS objective Gradient descent stochastic gradient descent 3

What s the mileage? Suppose we want to predict the mileage of a car from its weight and age Weight (x 100 lb) x 1 Age (years) x 2 Mileage 31.5 6 21 What we want: A function that can predict mileage using x 1 and x 2 36.2 2 25 43.1 0 18 27.6 2 30 4

Linear regression: The strategy Predicting continuous values using a linear model Assumption: The output is a linear function of the inputs Mileage = w 0 + w 1 x 1 + w 2 x 2 Learning: Using the training data to find the best possible value of w Prediction: Given the values for x 1, x 2 for a new car, use the learned w to predict the Mileage for the new car 5

Linear regression: The strategy Predicting continuous values using a linear model Assumption: The output is a linear function of the inputs Mileage = w 0 + w 1 x 1 + w 2 x 2 Parameters of the model Collectively, a vector Learning: Using the training data to find the best possible value of w Prediction: Given the values for x 1, x 2 for a new car, use the learned w to predict the Mileage for the new car 6

Linear regression: The strategy Inputs are vectors: x R d Outputs are real numbers: y R We have a training set D = { (x 1, y 1 ), (x 2, y 2 ),!, (x m, y m )} We want to approximate y as For simplicity, we will assume that x 1 is always 1. That is x = [1 x 2 x 3... x d ] T This lets makes notation easier y = f w (x) = w 1 + w 2 x 2 +! + w n x n = w T x w is the learned weight vector in R d 7

Examples y One dimensional input x 2 8

Examples y Predict using y = w 1 + w 2 x 2 One dimensional input x 2 9

Examples y Predict using y = w 1 + w 2 x 2 One dimensional input x 2 Two dimensional input Predict using y = w 1 + w 2 x 2 +w 3 x 3 10

What is the best weight vector? Question: How do we know which weight vector is the best one for a training set? For an input (x i, y i ) in the training set, the cost of a mistake is Define the cost (or loss) for a particular weight vector w to be Sum of squared costs over the training set One strategy for learning: Find the w with least cost on this data 11

What is the best weight vector? Question: How do we know which weight vector is the best one for a training set? For an example (x i, y i ) in the training set, the cost of a mistake is Define the cost (or loss) for a particular weight vector w to be Sum of squared costs over the training set One strategy for learning: Find the w with least cost on this data 12

What is the best weight vector? Question: How do we know which weight vector is the best one for a training set? For an input (x i, y i ) in the training set, the cost of a mistake is Define the cost (or loss) for a particular weight vector w to be One strategy for learning: Find the w with least cost on this data 13

Least Mean Squares (LMS) Regression Learning: minimizing mean squared error 16

Least Mean Squares (LMS) Regression Learning: minimizing mean squared error Different strategies exist for learning by optimization Gradient descent is a popular algorithm (For this minimization objective, there is also an analytical solution) 17

We are trying to minimize Gradient descent J(w) General strategy for minimizing a function J(w) Start with an initial guess for w, say w 0 Iterate till convergence: Compute the gradient of the gradient of J at w t Update w t to get w t+1 by taking a step in the opposite direction of the gradient Intuition: The gradient is the direction of increase in the function. To get to the minimum, go in the opposite direction w 18

We are trying to minimize Gradient descent J(w) General strategy for minimizing a function J(w) Start with an initial guess for w, say w 0 Iterate till convergence: Compute the gradient of the gradient of J at w t Update w t to get w t+1 by taking a step in the opposite direction of the gradient w 0 Intuition: The gradient is the direction of increase in the function. To get to the minimum, go in the opposite direction w 19

We are trying to minimize Gradient descent J(w) General strategy for minimizing a function J(w) Start with an initial guess for w, say w 0 Iterate till convergence: Compute the gradient of the gradient of J at w t Update w t to get w t+1 by taking a step in the opposite direction of the gradient w 1 w 0 Intuition: The gradient is the direction of increase in the function. To get to the minimum, go in the opposite direction w 20

We are trying to minimize Gradient descent J(w) General strategy for minimizing a function J(w) Start with an initial guess for w, say w 0 Iterate till convergence: Compute the gradient of the gradient of J at w t Update w t to get w t+1 by taking a step in the opposite direction of the gradient w 2 w 1 w 0 Intuition: The gradient is the direction of increase in the function. To get to the minimum, go in the opposite direction w 21

We are trying to minimize Gradient descent J(w) General strategy for minimizing a function J(w) Start with an initial guess for w, say w 0 Iterate till convergence: Compute the gradient of the gradient of J at w t Update w t to get w t+1 by taking a step in the opposite direction of the gradient w 3 w 2 w 1 w 0 Intuition: The gradient is the direction of increase in the function. To get to the minimum, go in the opposite direction w 22

We are trying to minimize Gradient descent J(w) General strategy for minimizing a function J(w) Start with an initial guess for w, say w 0 Iterate till convergence: Compute the gradient of the gradient of J at w t Update w t to get w t+1 by taking a step in the opposite direction of the gradient w 3 w 2 w 1 w 0 Intuition: The gradient is the direction of increase in the function. To get to the minimum, go in the opposite direction w 23

We are trying to minimize Gradient descent for LMS 1. Initialize w 0 2. For t = 0, 1, 2,. 1. Compute gradient of J(w) at w t. Call it J(w t ) 2. Update w as follows: r: Called the learning rate (For now, a small constant. We will get to this later) 24

We are trying to minimize Gradient descent for LMS 1. Initialize w 0 2. For t = 0, 1, 2,. What is the gradient of J? 1. Compute gradient of J(w) at w t. Call it J(w t ) 2. Update w as follows: r: Called the learning rate (For now, a small constant. We will get to this later) 25

We are trying to minimize Gradient of the cost The gradient is of the form Remember that w is a vector with d elements w = [w 1, w 2, w 3,! w j,!, w d ] 26

We are trying to minimize Gradient of the cost The gradient is of the form 27

We are trying to minimize Gradient of the cost The gradient is of the form 28

We are trying to minimize Gradient of the cost The gradient is of the form 29

We are trying to minimize Gradient of the cost The gradient is of the form 30

We are trying to minimize Gradient of the cost The gradient is of the form 31

We are trying to minimize Gradient of the cost The gradient is of the form One element of the gradient vector 32

We are trying to minimize Gradient of the cost The gradient is of the form One element of the gradient vector Sum of Error Input 33

We are trying to minimize Gradient descent for LMS 1. Initialize w 0 2. For t = 0, 1, 2,. 1. Compute gradient of J(w) at w t. Call it J(w t ) Evaluate the function for each training example to compute the error and construct the gradient vector 2. Update w as follows: (until total error is below a threshold) r: Called the learning rate (For now, a small constant. We will get to this later) One element of " 37

We are trying to minimize Gradient descent for LMS 1. Initialize w 0 2. For t = 0, 1, 2,. (until total error is below a threshold) 1. Compute gradient of J(w) at w t. Call it J(w t ) Evaluate the function for each training example to compute the error and construct the gradient vector 2. Update w as follows: The weight vector is not updated until all errors are calculated Why not make early updates to the weight vector as soon as we encounter errors instead of waiting for a full pass over the data? 41

Stochastic gradient descent Repeat for each example (x i, y i ) Pretend that the entire training set is represented by this single example Use this example to calculate the gradient and update the model Contrast with batch gradient descent which makes one update to the weight vector for every pass over the data 42

Stochastic gradient descent 1. Initialize w 2. For t = 0, 1, 2,. (until error below some threshold) For each training example (x i, y i ): Update w. For each element of the weight vector (w j ): 43

Stochastic gradient descent 1. Initialize w 2. For t = 0, 1, 2,. (until error below some threshold) For each training example (x i, y i ): Update w. For each element of the weight vector (w j ): Contrast with the previous method, where the weights are updated only after all examples are processed once 44

Stochastic gradient descent 1. Initialize w 2. For t = 0, 1, 2,. (until error below some threshold) For each training example (x i, y i ): Update w. For each element of the weight vector (w j ): This update rule is also called the Widrow-Hoff rule in the neural networks literature Online/stochastic algorithms are often preferred when the training set is very large May get close to optimum much faster than the batch version 46

Learning Rates and Convergence In the general case the learning rate r must decrease to zero to guarantee convergence The learning rate is called the step size. More sophisticated algorithms choose the step size automatically and converge faster Choosing a better starting point can also have impact Gradient descent and its stochastic version are very simple algorithms Yet, almost all the algorithms we will learn in the class can be traced back to gradient decent algorithms for different loss functions and different hypotheses spaces 47

Linear regression: Summary What we want: Predict a real valued output using a feature representation of the input Assumption: Output is a linear function of the inputs Learning by minimizing total cost Gradient descent and stochastic gradient descent to find the best weight vector This particular optimization can be computed directly by framing the problem as a matrix problem 48

Exercises 1. LMS regression can be solved analytically. Given a dataset D = { (x 1, y 1 ), (x 2, y 2 ),!, (x m, y m )}, define matrix X and vector Y as follows: Show that the optimization problem we saw earlier is equivalent to This can be solved analytically. Show that the solution w* is Hint: You have to take the derivative of the objective with respect to the vector w and set it to zero. 49