Data fitting. Ideal equation for linear relation

Transcription

1 Problem Most likely, there will be experimental error so you should take more than 2 data points, & they will not lie on a line so the ideal eqn above has no solution. The question is what is the best soln. Daniel Chan (UNSW) Lecture 37: Least squares Semester / 9 Data fitting Aim lecture: Sometimes we cannot solve the lin eqn Av = w, but nevertheless wish to find the best possible soln v. We use the theory of orthogonal projections to accomplish this. Motivating example Suppose we know that two physical variables y, t are linearly related say by y = α + βt for some unkown α, β R. We wish to determine α, β experimentally & plot data points (t 1, y 1 ),..., (t n, y n ) for distinct t 1,..., t n R. If all variables are perfectly measured then α, β can be computed by solving the Ideal equation for linear relation The soln to 1 t t n ( ) α = β gives the linear fn y = α + βt relating y & t. y 1. y n

2 Best approximation The following gives a new characterisation of orthogonal projections. Prop-Defn Let A M nm (R) & X = im A R n. Let w R n. 1 As x varies over X, x = proj X w is the unique vector which minimises w x. Consequently, proj X w is also called the best approximation to w in X. 2 x = proj X w is also the unique vector in X which satisfies A T x = A T w. Proof. 1) is just Pythogoras thm (ex). DRAW PICTURE 2) R n = X X so x = proj X w is the unique vector in X such that w x (im A) = (im (A T ) T ). The propn on orthogonal complements of kernels in lecture 36 = (im A) = ker A T. Hence x = proj X w is the unique vector in X such that A T (w x) = 0. This gives 2). Daniel Chan (UNSW) Lecture 37: Least squares Semester / 9

3 Least squares solution Cor-Defn Let A M nm (R) & w R n. A least squares solution to Av = w is any v R m which minimises the error term Av w. Equivalently, it is any v R m such that 1 Av is a best approximation to w in im A. 2 it is a solution to the normal equation A T Av = A T w. If ranka = m (i.e. the columns are lin indep) then A T A is invertible so the least squares soln is given uniquely as v = (A T A) 1 A T w. In particular, we obtain the projection formula proj im A w = A(A T A) 1 A T w. Proof. 1) & 2) follow from the previous propn since Av varies over im A as v varies over R m. Suppose now that m = ranka = ranka T A by propn lecture 36, so A T A M mm (R) is invertible. The projection formula follows from the fact that proj im A w = Av where v is any least squares soln. Daniel Chan (UNSW) Lecture 37: Least squares Semester / 9

4 Example E.g. Find the (least squares) line of best fit y = α + βt to the data points (t, y) = (1, 5), (2, 3), (3, 3), (4, 0). A We wish to find the least squares soln to the ideal eqn on page 1. Daniel Chan (UNSW) Lecture 37: Least squares Semester / 9

5 Example cont d Daniel Chan (UNSW) Lecture 37: Least squares Semester / 9

6 Curves of best fit One can further ask what is the a parabola of best fit y = α + βt + γt 2 to a set of data points & so forth. The general setup is as follows. Setup Consider data points (t 1, y 1 ),..., (t n, y n ) for distinct t 1,..., t n R. Let φ 1 (t),..., φ m (t) be R-valued fns of t. Curve of best fit Suppose y(t) has the form y(t) = m i=1 β iφ i (t) for some parameters v = (β 1,..., β m ) T R m. Let w = (y 1,..., y n ) T. The curve of best fit (wrt linear combns of φ 1,..., φ m ) is given by the least squares soln to Av = w where A = (φ j (t i )) ij M nm (R). E.g. Find the quadratic function form y = β 1 t + β 2 t 2 which best fits ( 1, 1), (1, 1), (2, 1). We wish to best solve Daniel Chan (UNSW) Lecture 37: Least squares Semester / 9

7 Example cont d Daniel Chan (UNSW) Lecture 37: Least squares Semester / 9

8 An application of QR-factorisation QR-factorisation is frequently used in numerical linear algebra (i.e. numerical algorithms that computers can implement to solve matrix problems). Numerical Problem Suppose that A M mn (R) is a matrix whose columns are almost linearly dependent. For example 1 1 A = ε for some small ε > 0. Then A T A will be close to non-invertible in the sense that det(a T A) is small. By Cramer s rule, small round-off errors in computing A T A may lead to large errors in solving the normal eqn A T Av = A T w. In our example above ( ) A T ε A = 3 + ε 3 + 2ε + ε 2 has determinant 2ε 2. Here if your computer rounded off ε 2 to 0, the determinant changes to ε 2 so the computer error in solving the normal eqns is roughly a factor of 2! Daniel Chan (UNSW) Lecture 37: Least squares Semester / 9

9 Application cont d Question How can you get around this? Lemma Let Q = (w 1... w n ) M mn (C) be an m n-matrix with columns w 1,..., w n 0. 1 The columns are orthogonal iff Q Q is diagonal. 2 In this case, Q Q is the diagonal matrix D = ( w 1 2 )... ( w n 2 ) so Q has a left inverse Q = D 1 Q i.e. Q Q = I n. 3 {w 1,..., w n } is orthonormal iff Q Q = I m. Proof. We think of Q as a 1 n-matrix with entries w 1,..., w n M m1 (C). We may also view Q as an n 1-matrix with entries w 1,..., w n M 1m (C). We then multiply these matrices of matrices to see the (i, j)-th entry of Q Q is w i w j = (w i w j ). All results now follow. A QR-factorisations sometimes helps as follows. Suppose A = QR is the QR-factorisation. Then the normal eqn is R T Q T QRv = R T Q T w R T Rv = R T Q T w Also, R T is invertible so this reduces to the better behaved Rv = Q T w since det(r) = det(a T A). Daniel Chan (UNSW) Lecture 37: Least squares Semester / 9