Lecture 3 Notes. Dan Sheldon. September 17, 2012

Transcription

1 Lecture 3 Notes Dan Shelon September 17, Errata Section 4, Equation (2): yn 2 shoul be x2 N. Fixe 9/17/12 Section 5.3, Example 3: shoul rea w 0 = 0, w 1 = 1. Fixe 9/17/12. 1 Review: Linear Regression Setup Training ata (x 1, y 1 ), (x 2, y 2 ),..., (x N, y N ), where x i, y i R (i.e., real numbers: 0.3, 1.56, π, etc. ) Hypothesis h w (x) = w 0 + w 1 x (linear function) Parameters w 0, w 1 : each ifferent value of parameters gives a ifferent hypothesis Goal: fin hypothesis h w that is best fit to training ata Cost function (aka loss function) Numerical measure of fit between hypothesis an training ata Higher cost worse fit Square error cost function (Gauss) cost(h w ) = (h w (x i ) y i ) 2 Substitute form of linear hypothesis h w (x) = w 1 x + w 0 into cost function: J(w 0, w 1 ) = (w 0 + w 1 x i y i ) 2 Simplification (for now): assume w 0 = 0 h w (x) = w 1 x. New cost function is J(w 1 ) = (w 1 x i y i ) 2 For any given training set, the cost function is a function of parameters only. 1

2 Example 1 (Assuming w 0 = 0 ). Consier the following training set The cost function is x y J(w 1 ) = (w 1 x 1 y 1 ) 2 + (w 1 x 2 y 2 ) 2 = (w 1 2) 2 + (2w 1 2) 2 = 5w w This is a quaratic function of w 1, so we can fin the minimum by optimization. 2 Illustration: Hypothesis vs. Cost Function Each hypothesis equate with numerical parameters Parameter space - set of all possible parameters To fin best hypothesis: minimze cost function over parameter space. 3 Derivatives: What You Nee To Know For a function f(x), enote the erivative of f by x f(x) (sometimes f (x)) Derivative = slope of the tangent line to f at x Derivative is equal to zero at a minimum of f(x) 2

3 Illustration: minima, maxima, local minima, convex function (bowl-shape) 4 Minimizing J(w 1 ) One way to fin a minimum (which works for linear regression, but not every problem) is to fin the erivative, set it equal to zero, an solve the resulting equation. Example 2 (Continuation of Example 1). To minimize J(w 1 ) = 5w w , set the erivative equal to zero an solve for w 1 : 0 = w 1 J(w 1 ) = 10w w 1 = 16 w 1 = 8 5 = 1.6. General Case: For the general problem, we can solve for w 1 in terms of (x 1, y 1 ), (x 2, y 2 ),... (x N, y N ). We nee the following fact (which you can verify if you know calculus): w 1 J(w 1 ) = w 1 (w 1 x i y i ) 2 = 2 (w 1 x i y i )x i = 2 (w 1 x 2 i x i y i ). (1) Then, we can set the erivative to zero an solve, to get 0 = 2[(w 1 x 2 1 x 1 y 1 ) (w 1 x 2 N x N y N )] w 1 (x x 2 N ) = (x 1 y x N y N ) w 1 = x 1y x N y N x x2 1 (2) We can apply this formula for w 1 to any training set to get the best fit line. It is our first ML algorithm! 5 Graient Descent But we want to minimize J(w 0, w 1 ), not J(w 1 ). For general ML problems, not always possible to minimize cost function by setting erivatives to zero (In this case it is possible, but laborious. See Equation (18.3) on p. 719 of R&N for the answer). Graient escent: simple an very broaly applicable algorithm to minimize any function of J(w 0, w 1,..., w ) of multiple variables. Requirement: be able to compute partial erivatives w j J(w 1,..., w ) Mathemetical efinition of algorithm ( = 2): 1. Initialize w 0, w 1 arbitrarily (e.g. w 0 = 0, w 1 = 0) 2. Repeat until convergence Implementation note: make upates simultaneously w 0 = w 0 α w 0 J(w 0, w 1 ) w 1 = w 1 α w 1 J(w 0, w 1 ) 3

4 1. Initialize w 0, w 1 arbitrarily 2. Repeat until convergence 5.1 Illustration In One Dimension. Repeat: 0 J(w 0, w 1 ) w 0 1 J(w 0, w 1 ) w 1 w 1 w 1 α 0 w 2 w 2 α 1 w 1 w 1 α w 1 J(w 1 ) 5.2 Illustration In Two Dimensions 4

5 5.3 Graient Descent for Linear Regression To solve the linear regression problem using graient escent, the only thing we nee to know are the partial erivatives for our cost function: Here they are: w j J(w 0, w 1 ) = w j (h w (x i ) y i ) 2 = (w 1 x i + w 0 y i ) 2, j = 1, 2. w j w 0 J(w 0, w 1 ) = 2 w 1 J(w 0, w 1 ) = 2 (h w (x i ) y i ) (h w (x i ) y i ) x i Note that we can rop the constant 2 an absorb it into the learning rate α Work these out on your own if you are comfortable with partial erivatives. Example 3 (Continuation of Example 1). Recall the training set: x y Initialize w 0 = 0, w 1 = 1, an take one step of graient escent. erivatives). First, compute h w (x 1 ) = 1, h w (x 2 ) = 2. Then, (Drop the factor of 2 in the partial So the new values (w 0, w 1) are w 0 J(w 0, w 1 ) = ( 1 2) + ( 2 3) = 8 w 0 J(w 0, w 1 ) = ( 1 2) 1 + ( 2 3) 2 = 13. w 0 = 0 (0.1)( 8) = 0.1 w 1 = 1 (0.1)( 13) = 0.3 5

6 A Derivatives: Optional Backgroun For common functions (polynomials, exponentials, log, etc.) there are rules to fin their erivatives. Here are a few of the most important rules. Linear Quaratic (important!): General polynomial: Linearity: Chain rule: Examples: x x = 1. x x2 = 2x. x xk = kx k 1. (af(x) + bg(x)) = a x x f(x) + b x g(x). x f(g(x)) = f (g(x)) x g(x) 1. x 3x = 3 2. x (3x + 4) = 3 3. x 3x2 = 6x 4. x (3x 4)2 = 2(3x 4) x (3x 4) = 2 (3x 4) 3 = 18x (another way) x (3x 4)2 = x (9x2 24x + 16) = 18x 24 A.1 Partial Derivatives For a function f(x, y) of two variables, the partial erivative with respect to x is enote xf(x, y). It is calculate by following the same rules, except y is treate as a constant. Example: x 3x2 y = x (3y)x2 = 6yx. Similarly, the partial erivative with respect to y, enote y f(x, y) is compute by treating x as a constant an ifferentiating with respect to y. Example: x 3x2 y = x (3x2 )y = 3x 2. For a function f(x 1, x 2,..., x ) of many variables, the partial erivative with respect to x i, enote x i f(x 1, x 2,..., x ), is compute by treating all variables expect x i as constants, an computing the erivative with respect to x i. 6