Multiple-Linear, Polynomial and Nonlinear Regression Basic Concepts (1)

Transcription

1 Multple-Lnear, Polnomal and onlnear Regresson Basc oncepts Let us assume that there s a set of data ponts of a dependent varable versus,, n, where,, n are n ndependent eplanator varables. A partcular model to be ftted to the data s of the form g, Kn,, K m where K m are m parameters of the model. The least-squares error approach s most often used to fnd the parameters of Equaton., The statstcal assumpton behnd the least-squares error method for parameter estmaton s that the measured value of the dependent varable has a determnstc and a stochastc part. The stochastc part s often denoted as an error, ε. g ± ε, Kn,, K It s further assumed that the orgn of ε s measurement error, whch s randoml dstrbuted. m a Multple-Lnear, Polnomal and onlnear Regresson Basc oncepts An nfnte number of measurements would be requred to obtan the true values of the parameters, K m.because a sample alwas contans a fnte number of measurements, the calculated parameters are alwas appromatons for the true values. The are denoted wth a crcumfle. Thus,, K m are the calculated values of the parameters and ŷ s the calculated estmate for the dependent varable. In the least-squares error approach, the estmates, K m are found so that the mnmze the followng functon: [ ] g, Kn,, K m where s the sum of squares of the errors.the partcular mathematcal technque of fndng the set of the parameter values that mnmzes the functon depends on the form of the functon g,. If the parameters appear n lnear epressons n the functon g n multplelnear and polnomal regressons, for eample, the mnmzaton can be carred out b solvng a sstem of lnear equatons the normal equatons. Often models where the parameters appear n nonlnear epressons can be transformed to lnear models b transformaton of varables.

2 Graphc nformaton for checkng the qualt of the ft. An assessment of the qualt of the ft of a partcular model and comparson between dfferent models s based on graphc and numerc nformaton. The measured and the calculated values of the dependent varable can be plotted versus f there s a sngle ndependent varable or versus, the pont number f there are several ndependent varables. The dstance between the epermental values and the calculated curve can serve as an ndcaton for the qualt of the ft. These dstances are amplfed usng the "resdual plot". In ths plot the model error s plotted usuall versus, where ε ŷ A random dstrbuton of the resduals around zero ndcates that the model represents correctl the set of data. A defnte trend or pattern n the resdual plot ma ndcate ether lack of ft of the model or that the assumed error dstrbuton for the data random error dstrbuton n s not correct. 3 umerc nformaton for checkng the qualt of the ft. The most frequentl used numerc ndcator of the qualt of the ft s the standard error of the estmate whch represents the sample varance, and gven b s m 4 Thus the sample varance s the sum of squares of errors dvded b the degrees of freedom where the number of parameters, m, s subtracted from the number of data ponts, and s a measure for the varablt of the actual values from the predcted ŷ values. Smaller varance means a better fr of the model to the data. It should be emphaszed that when the sample varance s used for comparson of dfferent models, the same ndependent varable transformed or non-transformed should be used n Equaton 4 for all the models. The varance s an un-scaled varable whch can take an value from zero to nfnt. onsequentl the varance alone cannot be used for judgng the goodness of ft between the data and a respectve model.

3 umerc nformaton for checkng the qualt of the ft. The lnear correlaton coeffcent R s often used to judge the qualt of the ft between the regresson model and the epermental data. The correlaton coeffcent represents the rato between the sum of squares about the mean due to regresson to the total sum of squares, and s obtaned b R where s the sample mean of the dependent varable. The value of R s bounded: R. If R s close to there s a strong correlaton between the varables, whereas a value close to zero ndcates a weak or no correlaton. onfdence ntervals on the parameter values are ver useful ndcators of the ft between the model and the data. The dscusson concernng the confdence ntervals s postponed after dscussng the soluton technques of the normal equaton 5 Smple Lnear Regresson The smplest eample of least squares appromaton s fttng a straght lne to a set of pared measurements or observatons,,,,,. Introducng ths model nto Equaton elds [ ] The crteron for optmalt requres that and Thus, 7 8 [ ] [ ] 9 3

4 After rearrangement Smple Lnear Regresson Usng ramer's rule to solve for and elds Eample. ttng a Straght Lne to Thermal onductvt Data Thermal conductvt of low-pressure gases can be farl well correlated, over small temperature ranges, wth a lnear equaton straght lne. A lnear equaton should be ftted to the thermal conductvt data of ar shown n Table and the approprateness of the lnear model should be assessed. Table. Thermal conductvt of Ar a o Temperature Thermal onductvt* 6 cal/s cm

5 EXEL soluton of Eample A o. 9 sum B -4 - SUMB4:B SUM4: 3*D3- B3*E3/A*D3-B3^ A*E3- B3*3/A*D3-B3^ D B4^ B5^ B^ E B4*4 B5*5 B* SUMD4:D SUME4:E The values are stored n column B, the values are stored n column, the values are calculated n column D and the values are calculated n column E. The respectve sums are calculated n row 3. In cells 4 and 5 the varous terms are ntroduced nto Equaton n order to calculate and. The numercal results obtaned are shown below. EXEL soluton of Eample A B D E 3 o sum

6 EXEL soluton of Eample 3 To prepare a resdual plot and to calculate the varance and the correlaton coeffcent addtonal columns must be defned A G H I J o. 9 sum calc $$4$$5 *B4 $$4$$5 *B5 $$4$$5 *B ŷ ε G4^ G5^ G^ SUMH4:H num 4-$$6^ 5-$$6^ - $$6^ SUMJ4:J - $$6^ SUMI4:I 4- $$6^ 5- $$6^ In column the estmated values are calculated. In column G the resduals ε used for preparng the resdual plot are evaluated. In column H the resdual values are squared to enable calculaton of the varance and n columns I and J the numerator num and the denomnator den of equaton 5 are calculated. ε den EXEL soluton of Eample 4 The followng addtonal epressons are needed to complete the calculatons Mean Degrees of freedom Varance A orrelaton oeff. B AVERAGEB4:B AVERAGE4: A- H3/7 I3/J A Mean Degrees of freedom Varance orrelaton oeff. B It can be seen that the correlaton coeffcent, R s ver close to one, thus t seems that the lnear model represents ecellentl the data. or further analss let's look at the followng plots. 6

7 EXEL soluton of Eample Measured and alculated Values of Thermal onductvt Thermal onductvt of Ar 7 Thermal ond.*^ Temp. deg. Ths plot also ndcates ver good ft. The calculated and epermental ponts are actuall ndscernble n ths plot. But n the followng resdual plot the errors are not randoml dstrbuted around zero ndcatng that the model can probabl be further mproved. EXEL soluton of Eample Resdual Plot Resdual plot..5. Resdual Thermal cond. *^6 7

8 8 Multple Lnear Regresson Multple Lnear Regresson Let us develop the equatons to be solved for the case of two ndependent varables where a lnear model s to be ftted nto a set of measurements or observatons,,,,,,,,. The lnear model s Introducng ths model nto Equaton elds 3 The crteron for optmalt requres that, and. Thus, 4 [ ] [ ] [ ] [ ] Multple Lnear Regresson Multple Lnear Regresson After rearrangement and brngng nto matr-vector form we get 5 or the general case of m ndependent eplanator varables the matr form of the normal equatons can be more easl obtaned b defnng the followng matrces 6 The normal equaton s defned 7 m m m m M M K M M M K K Y X Y X X X T T

9 Multple Lnear Regresson 3 onfdence Intervals Ths s a sstem of lnear equatons s solved for the m coeffcents, m. If there s no free parameter n the model thus then the frst column of the numbers one should be removed from the matr X and the frst element, should be removed from the vector. It should be emphaszed that n such case the number of parameters n the model s m nstead of m, when there s a free parameter. Soluton of the sstem of equatons 7 provdes estmates for the parameter values. The uncertant n these appromate parameter values can be estmates usng the defnton of the confdence ntervals ts a ts a,k where t s the statstcal t-dstrbuton value correspondng to the degrees of freedom and the % confdence selected, s s the standard devaton square root of the varance and a s the th dagonal element of the X T X matr. The 95% confdence ntervals are used the most often. m 8 onfdence Intervals onfdence ntervals are ver useful ndcators of the ft between the model and the data. A better model ft and more precse data lead to narrow confdence ntervals, whle a poor model ft and/or mprecse data cause wde confdence ntervals. urthermore, confdence ntervals whch are larger n absolute value than the respectve parameter values often ndcate that the model contans superfluous parameters/ eplanator varables. Table. t-values correspondng to 95%confdence and ν degrees of freedom ν t-value ν t-value ν t-value

10 Eample. Heat Evolved Durng Hardenng Of Portland ement Woods et al93 nvestgated the ntegral heat of hardenng of cement as a functon of composton. The ndependent varables represent weght percent of the clnker compounds: -trcalcum alumnate 3aO Al O 3, -trcalcum slcate 3aO SO, 3 -tetracalcum alumno-ferrte 4aO Al O 3 e O 3, and 4 - -dcalcum slcate 3aO SO. The dependent varable, s the total heat evolved n calores per gram cement n a 8-da perod. o : alculate the coeffcents of a lnear model representaton of as functon of,, 3, and 4, calculate the varance and the correlaton coeffcent R and the confdence ntervals. Prepare a resdual plot. onsder the cases when the model ncludes and does not nclude a free parameter. Eample. Matlab Soluton Data Input and ormal Matr % flename heat_hardenng.m clear, clc, format short g, format compact X[ ]'; Y[ ]'; YmeanmeanY; 3; % o. of data ponts npar5; % o. of parameters - wth a free parameter t_95.36; % t-value for 95%confdence nterval - wth a free parameter %npar4; % o. of parameters - no free parameter %t_95.6; % t-value for 95%confdence nterval - no free parameter eones,; X[e X]; % Add column of ones to the X matr wth free parameter AX'*X;

11 Eample. Matlab Soluton alculatons and Resdual Plot AnvnvA; %alculate the nverse of the X'X matr for confdence nterval calculaton BetaAnv*X'*Y; % Solve the normal equaton YcalX*Beta; % alculated dependent varable values sy-ycal'*y-ycal/-npar; % varance RYcal-Ymean'*Ycal-Ymean/Y-Ymean'*Y-Ymean; for :npar end % %resdual plot % ploty,y-ycal,'*' onf_nt,t_95*sqrts*anv,; %confdence ntervals ttle'resdual plot, Heat of hardenng problem' label'heat of hardenngmeasured' label'resdual' %orrelaton oeffcent Eample. umercal Results Wth a free parameter Parameter o. Beta onf_nt Varance orrelaton oeffcent.9838 o free parameter Parameter o. Beta onf_nt Varance orrelaton oeffcent or the frst case where the model ncludes a free parameter the value of the correlaton coeffcent R.9838 and the resdual plot suggest that the model s approprate. But all the confdence ntervals are larger n absolute value than the respectve parameter values, ndcatng that there are too man parameters terms n the model. In the model where there s no free parameter all the confdence ntervals are also satsfactor. Thus the lnear model wthout a free parameter s approprate. Phscal consderaton lead also to the concluson that free parameter s not needed n ths case.

12 Eample. Resdual Plot Generalzed Multple Lnear Regresson. General models of the form g 9 f, K f, KK m fm, K an be brought nto the form, whch s approprate for multple lnear regresson ' ' ' ' K m b transformng the varables ' g, f,, f, etc. ' K The normal 6 and 7 can be solved for the coeffcents,, m, after the values of ' are ntroduced nto the vector Y and ' j ntroduced nto the matr X. m In polnomal regresson the transformatons:, ' and ' used. In Redel's equaton log P / T log T T, for vapor pressure correlaton the transformatons; ' log P, ' / T, log T and 3 should be used. ' ' 3 m ' K ' T m

13 Eample 3. ttng a nd Order Polnomal Thermal onductvt Data A nd order polnomal should be ftted to the thermal conductvt data of ar shown n Table to whch a straght lne was ftted n Eample. The same Matlab program that was used for soluton of Eample can be used onl the contants of the X matr the Y vector,, npar and t_95 should be changed. T[ ]'; Y[ ]'; 9; % o. of data ponts for : X,:[T T^]; end npar3; % o. of parameters - wth a free parameter t_ ; % t-value for 95%confdence nterval - wth a free parameter Eample 3. MATLAB results Parameter o. Varance nd order polnomal Beta E-5 orrelaton oeffcent onf_nt e Parameter o. Varance Straght lne Beta orrelaton oeffcent onf_nt rom the soluton t can be seen that the resdual plot of the nd order polnomal representaton s randoml dstrbuted the varance and the confdence ntervals for the polnomal representaton are smaller than for the straght lne and R s closer to one. Thus, the polnomal represents the data correctl. 3

14 Eample 3. Resdual Plot Phscal Propertes orrelaton 4

15 Phscal Propertes orrelaton Ingham, H.; rend, D.G.; El, J..; "Thermophscal Propertes of Ethane"; J. Phs. Ref. Data 99,, 75 Temperature K Ideal Gas Heat apact Ideal Gas Heat apact Temperature K E E E E E E E E E E E E E E E E E E E E4 4.43E E4 4.39E E E E E E E E E E E E E E E E E E E4 Phscal Propertes orrelaton Ingham, H.; rend, D.G.; El, J..; "Thermophscal Propertes of Ethane"; J. Phs. Ref. Data 99,, 75. Temperature Vapor Pressure K Pa 9.7 rtcal Temperature K 3.53E 94.8 rtcal Pressure Pa 4.87E Trple Pt Temperature K 9.35E

16 onlnear Regresson If one or more of the parameters of the model are ncluded n nonlnear epressons the estmaton of the parameters cannot be carred out b solvng a sstem of lnear equatons. The most general approach to solvng a nonlnear regresson problem s b usng optmzaton programs to mnmze Equaton numercall whle changng the parameters, m. or ndvdual cases, more specfc smpler technques can be used. Eample 4. ttng Parameters to the Antone equaton The followng table presents data of vapor pressure versus temperature for benzene. orrelate the data usng the Antone equaton. o. : 9 Temperature, T Pressure, P Mm Hg Eample 4. Soluton The Antone equaton s a wdel used vapor pressure correlaton that utlzes the parameters A, B, and. It can be epressed b B log P A T Defnng and ntroducng Equaton nto Equaton gves n B A T 3 6

17 7 Eample 4. Soluton Eample 4. Soluton Dfferentatng wth respect to A, B, and and equatng to zero, eld 4 Ths s a sstem of three nonlnear algebrac equatons wth three unknowns. The MATLAB functon to be used when solvng ths problem as a sstem of three equatons follows. T T B A T T B A B T B A A n n n MATLAB functon for solvng Eample 4 as a sstem of three nonl MATLAB functon for solvng Eample 4 as a sstem of three nonlnear near equatons equatons flename fun_antone %Solvng for Antone equaton parameters as a sstem of equatons functon ffun_antone A; B; 3; T[ ]'; P[ ]'; YlogP; f; f; f3; for : ff-yab/t; ff-y-a-b/t/t; f3f3y-a-b/t*b/t^; end f, f ; f, f ; f3, f3 ;

18 MATLAB functon for solvng Eample 4 Ths functon can be used wth a man program whch appl, for eample, the multdmensonal ewton-raphson method see the man program of Eample 3 n the prevous chapter of sstems of nonlnear algebrac equatons. Startng from the ntal estmate A 6, B -7, and 5 the program converges to the soluton: A , B , and It should be mentoned that when solvng ths problem as a sstem of nonlnear equatons, good ntal estmates for the parameters should be provded otherwse most soluton methods wll dverge. An alternatve approach whch does not requre close ntal estmates nvolves converson of the problem to a sngle nonlnear equaton. or a specfed value of, the frst two equatons of the sstem 4 are lnear and can be solved for A and B. Then the thrd equaton of ths sstem s used for calculatng f. The MATLAB functon for carrng out ths calculaton follows. MATLAB functon for solvng Eample 4 as a sngle nonlnear equaton %flename fun_antone %Solvng for Antone equaton parameters as a sngle nonlnear equaton functon ffun_antone T[ ]'; P[ ]'; YlogP; :,[./T]; den*'*-sum^; AsumY*'*-sum*'*Y/den; B*'*Y-sum*sumY/den; f; for : ffy-a-b/t*b/t^; end Ths functon can be used n conjuncton wth the man programs for solvng sngle nonlnear equatons that are provded n the frst chapter. Usng the bsecton method for eample man program of Eample 3 n hapter startng from 5 and 35, t converges to the soluton n teratons. 8