Over Determined Algebraic Systems: Curve Fitting to Data

Transcription

1 Over Determined Algebraic Systems: Curve Fitting to Data Introduction. In this section we investigate algebraic systems that have more equations than unknowns. In general, these will not have a solution in the sense that one gets equality for each equation. Here we hope to be able to choose the unknowns so that the equations are as close to being satisfied as possible. For example, consider the following where there are two variables and three equations x + x =, x + x = 3 and 3x + x = 7. Here there is no solution because there are more equations and only two unknowns. Or, in matrix form, with the residual vector, r, inserted is r x 3 = r. 3 x 7 r 3 We will want to choose the solution x so that the "size" of the residual vector is a minimum. A very appropriate measure of "size" will be the sum of the squares of residual's components. r T r = r + r + r3. Such x are called the least squares solution to the overdetermined system. Applied Area. We will apply the above concepts to two areas. Both are related to fitting curves to a number of data points. There are many other problems that can be formulated as minimization problem, which are in many cases related to algebraic systems. Application to Business Forecasting. Consider a computer company which has recorded sales of 78, 85, 90, 96, 04 and 3 computers over the last six months: Table: Sales Data Month Computers Sold

2 They wish to make a prediction of the sales over the next six months. This will help them plan their production needs during this period. The data is increasing more or less in a linear fashion. Consequently, we are looking for a straight line that is "closest" to the data. An analytical way of saying this is that we are looking for y = mx + c, that is, the slope, m, and the y intercept, c, so that this line is "closest" to the graphed data. Once the m and the c are known, then the future sales can be predicted by putting the appropriate month into the x variable and computing the forecasted sales in the y variable. Sales * * * * * * Figure: Time Sales Data and a "Close" Straight Line In order to form the corresponding over determined system, note the components of the residual vector are just the vertical distances between the data points and the desired straight line. So, for each data point there is an equation, which has the two unknowns m and c. For the above data the six equations are m + c = 78, m + c = 85, m3 + c = 90, m4 + c = 96, m5 + c = 04 and m6 + c = 3. Or, the matrix form with the residual vector is

3 r 85 r m 90 r 3. = c 96 r4 04 r 5 3 r6 Application to Nonlinear Heat Diffusion. In the introductory models of heat diffusion via the Fourier heat law, the thermal conductivity was the constant of proportionality for the diffusion of heat. However, if the temperature varies over a large range, the thermal conductivity will not be a constant. Consider the following data for thermal conductivity as a function of temperature. Table: Thermal Conductivity Data Temperature Thermal Conductivity Inspection of this data indicates that it is generally shaped like a parabola whose equation is a second order polynomial a + bx + cx. We want to choose the three coefficients so that the graph of the polynomial will be closest to the data. In this problem there are 5 data points and three unknowns, and the corresponding equations are a + b000 + c000 =.000, a + b00 + c00 =.005, a + b400 + c400 =.00, a + b600 + c600 =.005 and a + b800 + c800 = Or, the matrix form with the residual vector is

4 r a r b = r c.005 r r 5 Model. The model has the form of Ax = d where A is an m by n matrix with m larger than n, that is, there are more rows or equations than the unknowns in the x vector. Since there are no exact solutions, we try to find x so that the residual vector in Ax = d - r is as "small" as possible. Definition. The vector x is called a least squares solution of the over determined system if and only if x is such that r T r =(d - Ax) T (d - Ax) is a minimum of all (d - Ay) T (d - Ay) Method. Fortunately, this solution in most cases has a very nice answer. If x is the least squares solution, then for all y (d - Ax) T (d - Ax) (d - Ay) T (d - Ay). () Let y = x + (y - x) and consider (d - Ay) T (d - Ay) = ((d - Ax) - A(y -x)) T ((d - Ax) - A(y -x)) = ((d - Ax) T - (A(y -x)) T ) ((d - Ax) - A(y -x)) = (d - Ax) T (d - Ax) - (A(y -x)) T (d - Ax) - (d - Ax) T A(y - x) + (A(y -x)) T (A(y - x)) (d - Ax) T (d - Ax) - ((y - x) T A T (d - Ax). () If the last term on the right side of () is zero, then the inequality in () must hold. This prompts the following definition and theorem which we have just established. Definition. A T Ax = A T d is called the normal equation associated with the least squares problem. Normal Equation Theorem. If A T A has an inverse, then the solution of the normal equation is also a solution of the least squares problem. Example. unknowns Consider the problem in the introduction with three equations and two

5 5 x 3 x = 3 with = = T A A and 3 8 = 3 = T 7 Ad. The solution of the normal equation is x = 3 and x = -7/3. An interpretation of this is with x = slope = m and x = y intercept = c, is that the straight line y = 3x + (-7/3) is closest to the three data points (x i, y i ) = (,), (,3) and (3,7). Implementation. First we use the Matlab/Maple symbolic procedure linsolve to find the solution of the normal equations for the business forecast. Then Matlab is used to find the thermal conductivity. Matlab/Maple Symbolic Code for the Business Forecast Model. EDU» A=[ ; ; 3 ;4 ;5 ;6 ] A = EDU» transpose(a) ans = EDU» ata = transpose(a)*a ata = 9 6

6 6 EDU» d = [ ]' d = EDU» atd = transpose(a)*d atd = EDU» lssol = linsolve(ata,atd) lssol = [ 34/5] [ 058/5] This means the slope m = 34/5 and the y intercept c = 058/5. The predicted sales at 9 months is m9 + c = 3. Matlab Code for the Thermal Conductivity EDU»format long (6 digits) EDU»A = [ 0 0; ; ; ; ] A = EDU»d = [ ]' d =

7 7 EDU»lssol = (A'*A)\(A'*d) lssol = EDU» ezplot(' *x *x*x',[0 800]) x ~~~ *x *x*x Figure: x Thermal Conductivity versus Temperature Assessment. The least squares linear approximation model requires that the data be in the form of a straight line. Other factors governing the effectiveness in economic models include unpredictable costs, attractiveness of items may increase or decrease rapidly, and new technologies. The thermal conductivity model may not be valid if there are changes of physical state within the desired range. In this case, the thermal conductivity will have jump discontinuities, and therefore, polynomial approximations will not be a good model. Not all models are in the form of polynomials. In many cases the models have the form of transcendental functions such as log or exponential functions. Here the least squares model must be altered to include these more complicated nonlinear effects. In another polynomial approximation, Lagrange polynomials, given by requiring the approximation to be exactly the value of a complicated function at a number of points. If there are a large number of points, then the Lagrange polynomial will have a high

8 8 order, and there will be many oscillations. By using least squares with the appropriate degree polynomial, which reflects the data, one can use many data points. Often the least squares method generates an algebraic problem that is prone to large numerical errors. In this case, one is advised to use the QR-factorization method to solve the algebraic problem. This is incorporated into the backslash command in Matlab as is illustrated below.» A\d ans = » lsquad = A\d % Solves for the quadratic approximation lsquad = » lslinear = A(:,:)\d % Solves for the linear approximation lslinear =.0e-003 *

9 9» resquad = d - A*lsquad % Computes the residual for the quadratic resquad =.0e-003 * » reslinear = d - A(:,:)*lslinear % Computes the residual for the linear reslinear = » resquad'*resquad % Minimum of the least squares function for quadratic ans = e-007» reslinear'*reslinear % Minimum of the least squares function for linear ans = e-006 % One can compare either r'*r, or the graphs of the approximating functions.» syms x» ezplot(lsquad() + lsquad()*x + lsquad(3)*x*x, [ ])» hold on» ezplot(lslinear() + lslinear()*x, [ ])» plot([ ], d,'*')

10 0 x / / x x Homework.. The sales of air conditioners during the months of January, February, March and April were 4, 6, 7 and, respectively. Find the linear least squares solution associated with this data. Use it to predict the sales in May. Would this be a good model to make predictions for the months of July or August?. Use the nonlinear thermal conductivity function to find the steady state temperature of a thin wire with no heat loss through the lateral sides? 3. Justify the steps leading from equation () to (). 4. Consider the data (-,0), (-,0), (,0), (,0) and (.,.4). Approximate the above data in two ways: (i). use Lagrange fourth order polynomial and (ii). use linear least squares. Graph both and compare them.