ES-2 Lecture: More Least-squares Fitting. Spring 2017

Transcription

1 ES-2 Lecture: More Least-squares Fitting Spring 2017

2 Outline Quick review of least-squares line fitting (also called `linear regression ) How can we find the best-fit line? (Brute-force method is not efficient) Calculus-based approach Matrix based approach Measuring goodness of fit Multilinear regression output depends linearly on two or more inputs 2

3 Quick review: Motivation Understand data! 3 Weight Predict new points Interpolate Extrapolate Miles/gallon Measure scatter in data / error in prediction

4 Weight Quick review: Least-squares criterion Miles/gallon 4 Pick the line that minimizes the sum of squared errors S r = n å i= 1 e 2 i = å( yi - b - mxi ) i= 1 Need to be aware of outliers but usually, least squares is fine n 2

5 Outline Quick review of least-squares problem How can we find the best-fit line? (Brute-force method is not efficient) Calculus-based approach Matrix based approach Measuring goodness of fit Multilinear regression 5 In book: Linear least squares (14.3), but I ll present it merged with generalized least squares (15.3)

6 Calculus based approach (Section in book) Want to minimize Sr, which depends on y-intercept and slope (here,called a0 and a1) S r n ( a, ) å 0 a1 = ( yi - a0 - a1xi ) i= 1 2 6

7 From section

8 8 Result from calculus-based approach Algebra gives some equations you can code up: This answer is for lines only; we can do similar calculation for other models ( ) x a y b x x n y x y x n m i i i i i i = - - = å å å å å

9 Approach 2: Matrix - based (example with 3 points) Model: y = m x + b 9 (0.7, 3.1) (2,2.5) (3,1) m b = 1 An important note: the order in which we write the equations down doesn t matter

10 Reminder (from HW7, and video lecture on MatrixDeterminant ) So far, all the Ax=b problems we ve solved have had square A (# equations = # unknowns) If we have a square matrix, we can invert it 10 (n x n) (n x 1) (n x 1) A x = b

11 A way to solve non-square matrix problems (without any proof it s best) We can invert square matrices only but, we can always multiply both sides of an equation by the same thing, so 11 (m x n) (n x 1) (m x 1) A x = b

12 Pseudo-inverse The form below is called the pseudo-inverse 12 It s very useful for data fitting, where we have problems that are over-determined (more equations / measurements than unknowns) Using vector calculus, it s possible to prove this solution minimizes the mean-squared error

13 Matlab syntax To solve A x = b, where A isn t square, we can just do: 13 x = A\b; And Matlab will figure out the best method! Or, if you want to look at the pseudo-inverse directly, use pinv P = pinv(a); Then, say x = P*b

14 So what? Is this just another way of getting the same answer? 14 No if we can set up a data fitting problem with any matrix A (not just straight line) we know how to solve it Generalized least-squares (15.3)

15 Outline Quick review of least-squares problem How can we solve it? (Brute-force method is not efficient) Calculus-based approach Matrix based approach Measuring goodness of fit Multilinear regression 15

16 Quick review: measuring location and spread of data (Chapra 14.1) Measures of location : what is a typical data value? Mean Median 16 Measures of spread : how dispersed are the values around the mean? Standard deviation and variance

17 Definition of standard deviation (SD) Find S t, summed squares of differences from mean: then calculate SD as S t s = y n å i= 1 2 ( y - y) St = n -1 where n-1 is referred to as the degrees of freedom. i 17 Variance is just the standard deviation squared SD has same units as data (so we can make statements like patient age was /- 2.3 years )

18 Back to fitting lines: measuring error in the fit With least-squares fitting, we are trying to minimize the sum of the squared errors: 18 n S r = å 2 e i = i=1 n å i=1 ( y i - a 0 - a 1 x ) 2 i. So it s natural to measure that. The standard error of fit is defined as: s y/ x = S r n - 2

19 Different measures of spread Sum of residuals to mean S t = y i - y å( ) 2 Sum of residuals to fit S r ( y y ) 2 å i - = FIT 19 If Sr is smaller than St, that represents the improvement due to the line fit

20 r 2 : Coefficient of Determination (used in HW) Calculate Sr, sum of squared residuals to model: S r = n å i= 1 2 ( y - y ) Calculate St, sum of residuals to mean of data: i FIT Calculate the change, normalized by St: r 2 = S t - S r S t S t = n å i= 1 2 ( y - y) i r2 shows what fraction of the data variability is explained by the model 20

21 Two example fits with R^2 values R^2 = 0.38 R^2 = r 2 = S t - S r S t Interpretation: closer to 1 is better. As data get closer to fitted line, Sr -> 0 and r^2 -> St/St=1.

22 20 19 Clicker What is r^2 in these cases? Red line = fit, blue = data 1) FIT = MEAN 100 2) ) f(x) f(x) f(x) x x x a) 1, 0, <0 b) 0, 1, <0 c) 0, 1, >0 d) Not enough info r 2 = S t - S r S t

23 Moral: R^2 isn t a perfect metric (there isn t a perfect metric!) In general, finding good performance metrics can be challenging! R^2 will be small if slope is small R^2 will change depending on how wide of a range of data (in x ) we have available 23 It s important to look at your data. Considering plotting the residual errors (data fit) to see if there is a pattern in the errors