Least-Squares Regression

Transcription

1 Least-quares Regression ChEn 2450 Concept: Given data points (x i, ), find parameters in the function f(x) that minimize the error between f(x i ) and. f(x) f(x) x x Regression.key - eptember 22, 204

2 Introduction: Regression to a Linear Function f(x) (x 2,y 2 ) (x 3,y 3 ) What we would like: y + x y 2 + x 2 y 3 + x 3 Problem: 3 equations (3 data points), but only 2 unknowns (a0, a). (x,y ) x Idea: minimize the error between f(x i ) and (x i ). A measure of error (sum of squared errors): (y f(x )) 2 +(y 2 f(x 2 )) 2 +(y 3 f(x 3 )) 2 (y x ) 2 +(y 2 x 2 ) 2 +(y 3 x 3 ) 2 To minimize error, we change a0 and a to minimize. et the slope 2(y x )( )+2(y x 2 )( )+2(y x 3 2x (y x ) 2x 2 (y x 2 ) 2x 3 (y x 3 ) To fit a linear polynomial to three data points: 3y 3 + (x + x 2 + x 3 ) 3y (x + x 2 + x 3 ) (x + x 2 + x 3 )+ (x 2 + x x 2 3) olve for a0,. 2 Regression.key - eptember 22, 204

3 Hoffman Linear Least-quares Regression Concept: Given data points (xi,yi), find parameters in the function f(x) that minimize the error between f(xi) and yi. y, f(x) (x i, ) f(x) x [ f(x i )] 2 - um of the squared errors. To minimize (error between function & data), we take partial derivatives w.r.t. the function parameters and set to zero. AUME f(x) is an n p -order polynomial: a k f(x) n p k0 2x k i a k x k n p j0 a j x j i 0 Only for polynomials n p k0 a k x k i 2 k 0... n p i - data point index k - polynomial coefficient index i - data point index j - dummndex k - polynomial coefficient index np+ equations for np+ unknowns (ak). Equations are linear w.r.t. ak. Linear Least quares regression. 3 Regression.key - eptember 22, 204

4 Example: n p (Linear Polynomial) a k Linear polynomial, np: f(x) + x Apply chain rule. x k i X nx g i n p j0 g i a j x j i 0 [ x i ] 2 g 2 i g i x i nx (2g i )( ) 2( x i ) k 0... n p X nx g i Here we divided the entire equation by -2. (why?) g i nx (2g i )( x i ) 2x i ( x i ) 2( ) [ x i ] 2( x i ) [ x i ] 0 0 [ x i ] [ x i ] x i x i ( + x i ) ( + x i ) x i 2 equations, 2 unknowns (a0, a) 4 Regression.key - eptember 22, 204

5 Example (cont d.) x i ( + x i ) ( + x i ) x i 2 equations, 2 unknowns. Let s put these in Matrix form... For 4 points, y + y 2 + y 3 + y 4 ( + x )+( + x 2 )+( + x 3 )+( + x 4 ) y x + y 2 x 2 + y 3 x 3 + y 4 x 4 ( + x ) x +( + x 2 ) x 2 +( + x 3 ) x 3 +( + x 4 ) x 4 tep : define the solution variable vector. a0 tep 2: define the matrix and RH vector. " P x i P x i P x2 i # P! P x i " P x i P x i P x2 i # a0 P! P x i Linear least squares regression for a linear polynomial. 5 Regression.key - eptember 22, 204

6 Example - Reaction Rate Constant rate constant ln(k) ln(a) " Pre-exponential factor k A exp Gas constant R8.34 J/mol-K RT y + x RT y ln(k), x RT, ln(a), P x i P x i P x2 i Activation energy Temperature recall: ln(ab)ln(a)+ln(b) # a0 log(k) (/s) b0 b Benzene diazonium chloride Cl ln(a) Ea data best fit /RT (mol/j) x 0 4 Chlorobenzene T (K) Cl + 2 k (/s) ote: we need to calculate A (pre-exponential factor) from a0. ow we are ready to go to the computer to determine a0 and a. 6 Regression.key - eptember 22, 204

7 Polynomial Regression & The ormal Equations (Alternative Formulation for Linear Least quares) p + x + a 2 x 2 + a 3 x a n x n One equation for each observation ( equations) Given >n p observations (x i, ), and a n p order polynomial, find a j. x x 2 x n p x 2 x 2 2 x n p x x 2 xn p A OTE: this is an overdetermined (more equations than unknowns) linear problem for the coefficients, a i.. a np y. y b A T A A T b ormal Equations Another form of linear least-squares regression. Example - linear polynomial: x x 2.. x p(x) + x y y 2. x x 2 x y x x 2.. x x x 2 x y y 2. y 7 Regression.key - eptember 22, 204

8 The Two are One... Consider each of the previous approaches for a first order polynomial. a k Direct Least quares x i x k i k0 k x i x2 i n p j0 + x i + ( x i ) 0, x i ( x i ) 0 a j x j i 0 x i x 2 i k 0... n p, x i x i A A T b A T A A T A x x 2.. x A T b x i x i x i x2 i b Typically most convenient for linear regression problems. y y 2 y 3. y Matrix Transpose Approach 8 Regression.key - eptember 22, 204

9 Example - Reaction Rate Constant rate constant ln(k) ln(a) Pre-exponential factor k A exp Gas constant R8.34 J/mol-K RT y + x RT y ln(k), x RT, ln(a), RT RT 2.. RT A Activation energy Temperature recall: ln(ab)ln(a)+ln(b) ln(k ) ln(k 2 ). ln(k ) b log(k) (/s) Benzene diazonium chloride Cl ln(a) Ea data best fit /RT (mol/j) x 0 4 A T A A T b Chlorobenzene T (K) ote: need to calculate A (preexponential factor) from a0. Cl + 2 k (/s) Regression.key - eptember 22, 204

10 Linear Least quares Regression in matlab BEFORE you go to the computer, formulate the problem as a polynomial regression problem! Do it manually - the way that we just showed. ee MATLAB code for previous example posted on class website. This is my favored method, and provides maximum flexibility! Polynomial regression: ppolyfit(x,y,n) polyval(p,xi) evaluates the resulting polynomial at xi. gives the best fit for a polynomial of order n through the data. if n(length(x)-) then you get an interpolant. if n<(length(x)-) then you get a least-squares fit. You still must get the problem into a polynomial form. 0 Regression.key - eptember 22, 204

11 The R 2 Value How well does the regressed line fit the data? R 2 ( f(x i )) 2 ( ȳ) 2 (xi,yi) - data points that we are regressing. f(x) - function we are regressing to. f(xi) - regression function value at xi. ȳ Average value of R 2 Perfect fit Regression.key - eptember 22, 204

12 onlinear Least quares Regression Assume f(x) has n parameters ak that we want to determine via regression. [ f(x i )] 2 a k 0, ame approach as before, but now the parameters of f(x) may enter nonlinearly! k... n Example: f(x) ax b a X b X 2x b i ax b i 2ax b i ln(x i ) X ax b i ax b 2 i 0 0 X x b i ax b i X x b i ln(x X g 2 i, g i ax X ax i X 2g i x b X 2g i ax b i ln(x i ) Two nonlinear equations to solve for a and b. Can we reformulate this as a linear problem? 2 Regression.key - eptember 22, 204

13 Kinetics Example Revisited um of squared errors. A exp Minimize w.r.t. A and Ea. gi 2, g i @ X apple 2g i X X apple A 2g i exp Ea 2 rate constant A 0 0 Pre-exponential factor k A exp exp 2 exp 2A exp T i exp Gas constant R8.34 J/mol-K RT A exp Activation energy A exp Temperature A exp A exp 2 nonlinear equations with 2 unknowns A,. We will show how to solve this soon! 3 Regression.key - eptember 22, 204