Chapter 3 Response Surface Approximations

Transcription

1 Chapter 3 Respose Surface Approxmatos Most optmzato algorthms that are use for solvg aaltcal egeerg optmzato problems are sequetal ature. hat s, the objectve fucto ad costrats are evaluated at oe pot at a tme, ad the values at that pot, as well as prevous desg pot cotrbute to a decso o where desg space to move to for the ext evaluato. Whe the objectve fuctos ad (or) the costrats are evaluated b expermets rather Whe the objectve fuctos ad (or) the costrats are evaluated b expermets rather tha b aaltcal evaluatos, there s usuall a cetve to perform the expermets batches rather tha sgl. Oe reaso for batchg expermets s that most requre set-up tme, advace plag ad reservatos of expermetal facltes or techcas. Aother reaso s that expermetal errors make t dffcult to terpret the results of a sgle expermet. Whe a batch of expermets s performed, errors oe or two expermets ted to stad out. Duplcatg expermets for detcal omal codtos permts us to estmate the magtude of expermetal scatter due to errors ad varablt the propertes of the tested desgs. Fall, some of the expermetal scatter ca be averaged out b performg a large umber of expermets. Because of these advatages of rug expermets batches, expermetal optmzato has followed a dfferet route tha aaltcal optmzato. he stadard approach s to use a optmzato strateg that s based o the results of a batch of expermets. O the bass of the expermets, we costruct approxmatos to objectve fuctos ad/or costrats ad perform optmzato o the bass of these approxmatos. I most cases, the optmum obtaed s the tested, ad f satsfactor results are obtaed the desg procedure s termated. I some cases, the optmum s used as the cetral desg pot for a ew batch of expermets, ad the process s repeated oce or twce. hs process s sometmes called sequetal approxmate optmzato. However, because of the cost ad tme assocated wth coductg expermets, t s rare that the process s terated to covergece. Whe aaltcal calculatos were mostl based o closed form solutos or umercal models that requred mmal modelg ad computatos, the dfferece betwee aalss ad expermets was ver clear. However, toda umercal evaluatos of objectve fuctos ad costrats ofte share ma of the propertes of expermetal evaluatos. Frst, umercal models such as fte elemet structural models requre substatal vestmet of tme to set up ad debug. Furthermore, the evaluato of such models ma requre large computatoal resources, so that the cost of umercal smulato ma be comparable to the cost of expermets. Secod, wth aaltcal smulatos based o complex umercal models, ma sources of ose are ofte foud the results of umercal smulatos. hese cludes roud-off errors as well as errors due to complete covergece of teratve processes. Addtoall, 8

2 umercal smulatos are usuall based o dscretzato of cotua, ad the accurac of ths dscretzato depeds of the shape of the domas beg dscretzed. For example, whe stress aalss s performed a elastc bod usg fte-elemet dscretzato, the dscretzato error does ot chage smoothl wth the shape of the bod, because the umber of fte elemets ca var ol teger cremets. hese growg smlartes betwee aaltcal smulatos ad expermets create cetves to ru aaltcal smulatos batches ad use sequetal approxmate optmzato. Addtoall, the growg avalablt of parallel computers also provdes cetves for rug aaltcal smulatos batches. Fall, umercal smulatos are ofte ru wth software packages that are dffcult to coect drectl to optmzato programs. Approxmate sequetal optmzato provdes a mechasm for rug these software packages a stad-aloe mode ad coectg the optmzato programs to the approxmato. 3. Fttg a Approxmato to Gve Data Selectg the pots desg space where expermets are to be performed s possbl the most mportat part of obtag good approxmatos to sstem respose quattes that ma be used computg objectve fuctos ad costrats. However, ths pot selecto turs to be a dffcult optmzato problem tself, ad wll be dscussed the ext chapter. I ths secto we wll address the more modest questo of how to obta the best approxmato to a gve a set of expermets. We deote the respose fucto that we wat to approxmate as, ad assume that t ca be approxmated as a fucto of the desg varable vector x ad a vector of parameters, that s ˆ( x, ) (3..) where ŷ represets the approxmato, ad the error. We have the data from expermets at desg pots x ( x, ) (3..) ad we wat to fd the parameter vector that wll best ft the expermetal evdece. I ma o-desg applcatos, ŷ represets a umercal model such as a fte-elemet model, ad represets a set of phscal parameters such as masses ad stffesses. he process of fdg the values of the phscal parameters that best ft the expermetal results s called sstem detfcato. I desg applcatos, the vector of parameters ofte does ot have a phscal meag. Rather, we select some fuctoal represetato for ŷ, wth represetg some coeffcet to be determed so as to ft the data well. For example, wth a sgle desg varable x, a lear approxmato s of the form x (3..3) whle a ratoal approxmato ma take the form (3..4) x 9

3 he process of fdg to best ft the data s called regresso, ad ŷ s called a respose surface. he most commol used measure of the error the approxmato s the root-mea-square (rms) error e ˆ(, ) rms x. (3..5) Other measures that are ofte used are the average absolute error eav ˆ( x, ), (3..6) ad the maxmum error e max ˆ ( x, ). (3..7) max he populart of erms as a measure of the error s probabl due to the fact that t s ofte easer to estmate the that mmzes e rms tha the that mmzes the other two measures. However, there s also theoretcal justfcato for usg t. If the form of the model ŷ s exact (that s the true fucto s of that form) ad the error s ormall dstrbuted, wth o correlato betwee data pots ad all havg the same stadard devato, the mmzg e rms wll provde the best lear ubased estmate (BLUE) of. he value of that would be foud f we had a fte umber of expermets s sometmes thought of as the `true' value of the parameter vector. Wth a fte umber of expermets we obta ol a estmate of ths 'true' value, whch we deote as b. Estmatg the value of b that mmzes the rms error s eas for cases where ŷ s a lear fucto of, that s ( x), (3..8) Where (x) are gve shape fuctos, usuall moomals. For example, for the lear approxmato, Eq. 3..3, x. (3..9) Let us deote as b a estmate to that mmzes e rms for the gve expermets. he dfferece betwee the model ad the jth expermet, e, s e j j j j b ( x ), (3..0) or vector form, e= - Xb. (3..) Note that the (I,j) compoet of the matrx X s j ( x ). he rms error s the rms average of the compoets of e erms e ee, (3..) So that mmzg e rms s equvalet to mmzgee. Usg Eq. 3.. we have ee ( Xb) ( Xb) Xbb X b X Xb. 30

4 (3..3) o fd the vector b that mmzes e rms we dfferetate eewth respect to the compoets of b ad set the dervatves to zero. It s ot dffcult to check that we get X Xb X. (3..4) Equato 3..4 s a sstem of equatos called the ormal equato, ad we ca wrte ts soluto of ths equato as ( b X X) X. (3..5) However, the ormal equato s ofte ll-codtoed, especall whe s hgh. o avod some of the effects of the ll codtog we ca formulate the problem a slghtl dfferet form. Cosder the sstem Xb. (3..6) If we could solve ths equato exactl, the from Eq we see that the rms error wll be zero. However, ths sstem has equatos for the ukows, wth geeral larger tha, so that geeral, we caot fd a exact soluto. hat s, a vector b wll ot satsf Eq exactl, but stead there wll be a vector of resduals (dffereces betwee the left sde ad the rght sde of the equato). he soluto of the ormal equato s the vector b that mmzes the sum of the squares of the resduals. However, stead of solvg the ormal equatos, there are umercal methods, such as the QR decomposto, that solves for Eq drectl for the least square soluto, ad these are usuall more umercall stable tha umercal solutos of the ormal equatos. o mprove umercal stablt, t s also recommeded to traslate ad scale all the varables so that each chages the rage (-,). he vector of errors left after the least-squares ft s deoted ase r, ad the sum of the squares of the errors remag after the ft s deoted as SS e. Wth a bt of algebra we ca show that SS ee b X. (3..7) e r r 3

5 Example 3.. Ft a straght le bo b x to the followg three measuremets of. (0) = 0, () =, () = 0. Calculate frst a least square ft, ad the compare to a ft that wll mmze the maxmum error, ad to oe that wll mmze the average absolute error. If we wrte the three equatos that we would lke to satsf for the three pots, Eq we If we wrte the three equatos that we would lke to satsf for the three pots, Eq we get (0) b 0 () b 0 0 () b 0 b b 0 (3..8) So that the matrx X ad the vector are gve as 0 0 X, (3..9) 0 So that X X, (3..0) Ad 0 X 0 (3..) 0 We ca ow wrte the ormal equato, X Xb X, 3b0 3b 3b0 5b ad solve t to obta b 0 = /3 ad b = 0. So that our least-square ft s the le = /3. Wth two pots at = 0 ad oe pot at =, ths seems to be a sesble choce. he errors (resduals) at the three pots are e = /3, e = /3, ad e 3 = /3. So the sum of the squares of the errors s /3, ad the rms error s e rms (3..) We ca obta the same result also from Eq sce, b /3,0, b X /3, (3..3) the SS / 3 / 3, e SS / (3..4) e rms e B comparso, the average ad maxmum errors are 3

6 e , av emax (3..5) I geeral, t s dffcult to fd the respose surface that mmzes the maxmum error ad the respose surface that mmzes the average error. However, for ths smple example ths ca be easl doe. We expect that lke the le that mmzes the rms error, the les that mmze the other two errors would be horzotal les of the form c, 0c. o mmze the maxmum error, t s obvous that we must have c = 0.5, whch results a maxmum error e max I ths case ths s also the average error ad the rms error because all three pots have the same 0.5 error. o mmze the average absolute error, we ote that for the rage c [0,] the average absolute error wll be e c ( c) c c av, 3 3 (3..6) so that the mmum average error s acheved for c = 0. I that case the le s gve as = 0, the average error s the smallest at /3, but erms / , ad e max. he three cases are show Fg. 3.. For more complex examples, we should use computer software to perform the process of fttg the respose surface to the data. he followg example shows results for the softdrk-ca problem. Example 3.. Soft-Drk Ca Desg: A soft-drk compa has a ew drk that the pla to market, ad ther prelmar research has determed that the cost to produce ad dstrbute a cldrcal ca s approxmatel gve as C 0.8V 0 0.S, where C s the cost cets, V 0 s the volume flud ouces, ad S s the surface area of the ca square ches. he also determed that the ca sell the drk cas ragg from 5 to 5 ouces, ad the the prce P cets that ca be charged for a ca s estmated as P.5V 0 0.0V 0. Based o ther past experece the wll cosder ol a ca wth dameter D betwee.5 ad 3.5 ches, ad ther market research has show that soft-drk cas have to have a aspect rato of at least.0 to be eas to drk from. hat s, the heght H of the ca has to be at least twce the dameter. We express the volume ad the surface area terms of the dameter ad the heght of the ca V 0.5D /.8, S (0.5D ) DH, 0 H where the.8 factor s used to covert from cubc ches to flud ouces. he proft per ca, p ma be the wrtte as c p PC DH DH D DH c (0.5 ), 33

7 Feld tests are coducted ad resulted the followg 9 data pots, show as (dameter, heght, proft): (.8, 3.6, 5.9), (.4, 4.8, 3.), (3., 6.,.), (.5, 4.6, 4.7), (., 5.8,.0) (.7, 7., 0.9), (.4, 5.6, 4.9), (., 6.8,.5), (.6, 8.,.4) We wll assume that the equatos are more accurate tha the feld test results (how ca that be possble?). From the equatos, the correspodg trads are.8, 3.6, ), (.4, 4.8, 3.074), (3.,6.,.883), (.5, 4.6, 4.748), (., 5.8,.963) (.7, 7., 0.854), (.4, 5.6, 4.9), (., 6.8,.458), (.6, 8.,.38) Ft a quadratc polomal to the proft per ca usg the expermetal 9 data pots, Repeat wth a ft based o the 'true' cost. he quadratc polomal wll be of the form. p c b bd b3h b4d b5dh b6h (3..7) So that from the data we wll get 9 equatos for the sx coeffcets. he frst equato obtaed for the expermetal data s b.8b 3.6b3 3.4b4 6.48b5.96b6 5.9, (3..8) ad the last oe s b.6b 8b3 6.76b4 0.8b5 64b6.4, (3..9) Solvg the ormal equato ad substtutg the coeffcets to Eq we get p c D 0.30H 0.358D.00DH H. (3..30) I comparso, f we use the true values for the proft, we get D 0.675H 0.96D.09DH 0.5H. (3..3) p c 34

8 Note that seve of the e coeffcets are farl close, wth about 0 percet of oe aother. However, the coeffcet of H ad of H are qute dfferet, dcatg that these coeffcets are less mportat for fttg the data. hs does ot mea, however, that these coeffcets ma ot 35

9 affect predctos at other pots besde the data pots. We are thus wared that these coeffcets ma eed specal treatmet. he proft-per-ca cotours based o the expermetal data are show Fg. 3., ad the oes based o the 'true' values are show Fg

10 3. Estmatg the Accurac of the Respose Surface I most applcatos, the respose surface approxmato we costruct o the bass of gve data s teded for predcto of respose at other desg pots, tpcall for mprovg the desg. herefore the ultmate test of the respose surface s how well t predcts the respose at other pots of terest. However, f the respose surface does ot approxmate the respose well eve at the data pots, we caot expect t to approxmate well other pots desg space. he most mmedate measures of the accurac of the ft to the data are the varous errors dscussed earler, the rms error, the average absolute error ad the maxmum error. We deote as ^ the estmate of the respose surface for the respose, that s (see Eq b ( x), (3..) ad use to deote the values of the respose surface at the data pots ad e the error vector at these pots (vector of resduals). It s eas to check that the Xb, e ˆ. (3..) he average ad maxmum errors are the eav e, emax max e. (3..3) he sum of the squares of the remag errors, SSe s gve b Eq. 3..7, ad we ca calculate from t the rms error, erms as SSe erms. (3..4) However, ths calculato of rms error s qute msleadg f we wated to use t to assess the accurac of the respose surface. hs becomes clear f we ote that f the umber of data pots s equal to the umber of coeffcets,, the the respose surface wll pass through the data pots, ad the error wll be zero. We certal do ot expect that the error wll be zero at other pots, ot cluded the data. I fact, fttg a respose surface to a umber of pots equal to the umber of coeffcets (kow as saturated desg) s kow to ofte lead to poor approxmato, especall whe there s ose the data. A mpressve bod of theoretcal work has bee doe for the case where the ose the data s radom wth ormal dstrbuto wth zero mea ad stadard devato of, ad where the ose at oe pot s ucorrelated wth the ose at other data pots (e.g., Mers ad Motgomer, 995). O the bass of these assumptos we ca estmate the stadard devato of the ose the measuremets, whch also serves as a estmate of e rms at all the pots the rego of terest. A ubased estmate ˆ for ths stadard devato s gve as SSe ˆ, (3..5) 37

11 ad s kow as the stadard error. If ths estmate of the rms error s larger tha we ca tolerate for predctg values of the respose at caddate desg pots, we coclude that our respose surface s adequate. I ths case we must chage the form of the respose surface to tr to ft the data better. he smplest approach s to add terms to the polomal approxmato. If we used a lear polomal to start wth, we ma wat to go to a quadratc. If we used a quadratc, we ma wat to use a cubc. Ufortuatel, whle we ca alwas mprove the ft to the data b creasg the umber of terms the respose surface, t s ot clear that these gas wll traslate to gas predctg the respose surface at other pots. As we add coeffcets, we ru the dager of `overfttg' the data. hs dager s partcularl acute whe the data cotas substatal amout of ose. As we crease the umber of coeffcets we ma be fttg the ose rather tha the uderlg respose. hs dager s captured b Eq As we add more terms we expect to decrease the umerator, but the deomator wll also decrease. Aother measure related to the rms error s obtaed b calculatg the varato of the data from ts average. (3..6) he varato from the average s deoted as SS ad s gve as. (3..7) SS Smlarl the varato of the respose surface ŷ from the same average s deoted as SS r so that ˆ SSr. (3..8) he rato of the two, deoted R, measures the fracto of the varato the data s captured b the respose surface SS r SSe R, (3..9) SS SS where the last equalt s left to the reader as a exercse. As we add more coeffcets to the polomal, we reduce crease capabltes of our model R whch s gve as SSe Ra R SS SSe ad therefore R. However, as oted before, ths does ot mea that the predcto mprove. For ths reasos, there s a adjusted form of If the adjusted value,. (3..0) R, decreases as we crease the umber of coeffcets, t s a a warg that we ma be fttg the data better, but losg o predctve capablt. he ol true test to the predctve capabltes of the respose surface s evaluatg t at pots ot used ts costructo. hs s doe ofte b practtoers. However, because addtoal tests or umercal evaluatos of are ofte expesve, t 38

12 s worthwhle to look for was of checkg predctve capablt wthout performg addtoal evaluatos at ew pots. If the umber of pots used for the ft s substatall larger tha the umber of coeffcets, leavg out a few pots wll ot chage much the qualt of the ft. We ca therefore leave out a few pots, ft the respose surface to the remag pots ad check the error at the left-out pots. We ca the replcate the procedure wth other pots, so that each pot s left out oce, a procedure kow as cross valdato. Whe the cost of repeatg the ft s small (as t s for respose surface techques), cross valdato s usuall doe leavg oe pot at a tme. he procedure goes also b the ame of PRESS (for predcto error sum of squares), a ame that comes from usg the sum of the squares of the predcto errors as a measure of the predctve accurac of the ft. We deote the vector of errors as e p. It s possble to obta e p wthout actuall performg all of these fts. Whe ol oe pot s left out, t ca be show that the th compoet of e p s related to the th compoet of e r as er e p, (3..) E where E s the dagoal matrx of the so-called dempotet matrx E X ( X X ) X. (3..) It should be oted that the matrx X X s ofte ll codtoed, especall for large problems ad the the calculato of the matrx E from Eq. 3.. s ot ver relable. However, eve the drect calculato of e p b performg the fts, s usuall less expesve tha carrg out addtoal expermets order to test the accurac of the respose surface. Example 3.. A respose quatt s a fucto of a sgle varable x, ad has evetuall bee detfed to follow the smple relatoshp x. However, the frst set of measuremets that were take are gve the four (x, ) data pars (-,-.5), (-,-.5), (,.5), (,.75). From theoretcal cosderatos ou also kow that ( 0) 0. Ft a lear fucto ad a quadratc to the data, compare the two fts ad see how the model the true fucto. Lear ft: he respose surface s of the form b b x, ad usg the pot (0,0) as the thrd pots we have.5.5 X 0, 0. (3..3).5.75 So that 5 0 X X 0 0, 0 X 9.5. (3..4) So that we ca solve for b as 39

13 X X 0 0., 0 b ( X X ) X 0.95, (3..5) ad the lear respose surface, ad the error vector are ˆ 0.95 x, e 0.35, 0.575,0,0.35, 0.. (3..6) 0. 0 We ow calculate the sums of the squares, ad use Eqs ad 3..0 to obta 4 e a SS 9.5, SS , R , R (3..7) 3 hs lear ft appears to be qute satsfactor, wth the errors beg e 0.337, e 0.7, e (3..8) rms av max he estmate of the stadard devato of the ose (stadard error) the data from Eq s ˆ SSe 0.896, ˆ , (3..9) whch s le wth the actual fucto beg x, sce the errors we troduced the data are 0.5 ad 0.5. b b x b x Quadratc model: he respose surface s of the form 3 ad we get X 0 0, 0, (3..0) ad X X 0 0 0, X 9.5, (3..) We solve for b to obta 0.07 b X X X 0.95, (3..) So that our approxmato s x x, e 0.43,0.5, 0.07, 0.378,0.07, (3..3) We aga calculate the sums of the squares to obta SS 9.5, SS e , 4 R 0. 94, R a , (3..4) We see that there s ol a small mprovemet R, ad that R s actuall poorer, whch s a dcato that we have ot gaed a predctve capabltes b addg the quadratc terms. he error measures for the quadratc model are e 0.35, e 0.9, e, (3..5) rms av max 0.5 a 40

14 hese have ot chaged much compared to the lear model, wth small mprovemets the rms ad maxmum errors, ad small crease the average error. he estmate of the stadard devato of the ose the data (stadard error) from Eq s SS e ˆ 0.643, ˆ 0.54, (3..6) whch s a small crease over the lear case. hs crease s aother dcato that the quadratc approxmato s ot better tha the lear approxmato. he two approxmatos are compared Fg I Example 3.. we compared a lear model to a quadratc model. However, we do ot have to lmt ourselves to models that have all the terms up to a partcular order. For example, we ca cosder a quadratc model wthout the costat term, b x b x. Smlarl, wth two varables, t s commo for people to cosder a model of the form b bx b3 x b4 xx.hs model does ot clude quadratc terms x ad x, but t cludes the `teracto' term x x. We ca stpulate such a partal model o the bass of some kowledge of the behavor of the true fucto. However, most ofte such partal models are created b dscardg terms wth coeffcets whch caot be accuratel estmated o the bass of the data. Such coeffcets do ot have much effect o the accurac of the ft of the gve data, ad leavg them the model ca reduce ts predctve qualt for desg pots where these coeffcets have large effect o the predcto. I trg to ft a respose surface to the proft per ca, Example 3.., we observed that some of the coeffcets were sestve to small dffereces the data. We ca detf these coeffcets wthout chagg the data b estmatg the stadard devato of the coeffcets. Let us frst troduce the covarace matrx hs covarace matrx s defed as b of the vector of coeffcets b. 4

15 ( ) ( ) b be b be b (3..8) hat s, due to ose we get varous vectors b b fttg the data from several sets of expermets ad E(b) s the expected value (or average over ver large umber of expermets) of b ad b-e(b) s the devato of b from ts expected value. he the covarace matrx s the expected value of products of varous compoets of the dfferece. I partcular, the dagoal terms of the matrx are b defto the squares of the stadard devatos of the compoets of b, whle the off-dagoal terms are a measure of the correlato betwee compoets. It s possble to show that X X b, (3..9) so that wth our estmate ˆ of (see Eq. 3..5), we ca estmate the stadard devatos of the dvdual terms. he estmate of the stadard devato of the compoets of b s called the stadard error, se, so that ^ X X, se( b ) (3..30) A useful measure of the accurac of a compoet of b s the estmate of the coeffcet of varato, c, whch s the stadard devato of the compoet dvded b the absolute value of that compoet. he coeffcet of varato of the th compoet c s therefore the stadard error dvded b the compoet c se( b ), (3..3) b I ma respose surface procedures the quatt whch s used to assess the eed for a coeffcet s the verse of c, whch s called the test statstc, or the t-statstc. he coeffcet of varato s used a strateg called backward elmato (Mers ad Motgomer, p. 650). I ths strateg, we elmate the coeffcet wth the largest coeffcet of varato, perform aother regresso wth the remag terms, elmate the coeffcet wth the hghest coeffcet of varato, ad so o. We ca stop oce the coeffcet of varato of the remag terms s small eough, or we ca use a measure such as Ra to dcate to us whe elmatg addtoal terms s hurtg us. hs s llustrated b applg ths strateg to Example 3... Example 3.. Use backward elmato, startg wth the quadratc model of Example 3.., to fd the model wth the hghest value of R a. Cofrm our cocluso b a recalculatg coeffcets for a slght perturbato of the data at x to ( ). 35. For the quadratc model Example 3.., we foud that X X 0 0 0,, ˆ 0.643, (3..3) From ths we calculate 4

16 X X 0 0, b (3..33) he vector coeffcets for the quadratc model was Eq we get b 0.07, 0.95, , so from c 3.35, c 0.76, 3.56, c (3..34) Note that two of the coeffcets of varato are ver large. A value larger tha oe dcates that the stadard devato of the coeffcet s larger tha the value tself, so that we have ver lttle cofdece the value obtaed from the regresso. he coeffcet of varato assocated wth the costat term b s the largest oe, so we elmate t. hs leads to the partal quadratc model b x b x. For ths model we get X 0 0, 0, (3..35) Ad 0 0 X 9.5 X, X (3..36) We solve for b to obta 0.95 b X X X, (3..37) so that the approxmato s ˆ 0.95x x, e 0.6,0.597, 0, ,0.88, (3..38) We calculate the sums of the squares 9.5, SS 0.55, R , R SS e a (3..39) Comparg to the quadratc model Example 3.., we ote a substatal mprovemet Ra whch for the quadratc model was We also calculate a ew ˆ as SSe ˆ = ^ (3..40) 3 whch s also a mprovemet over the quadratc model. Next we calculate the ew covarace matrx 43

17 0 0 X X, 0 34 he ew coeffcets of varatos are the gve as b, (3..4) c 0.47, c 3.33, (3..4) We stll have a large value of c, so to cotue the process of backward elmato, we eed to elmate the b coeffcet, assocated wth the x term. However, we kow that we wll get the lear model back aga, because Example 3.., whe we ftted a lear model, ths model. Furthermore, both ˆ (0.4354) ad elded ˆ 0.95x R a (0.969) for the lear model obtaed Example 3.. are slghtl feror to our complete quadratc model. hs dcates that takg oe more step the backward elmato wll cofrm that the complete quadratc model s the best. Next we check how the coeffcets chage f we chage () =.35. If we repeat the quadratc model, we fd that we get ŷ = x x. We see that the small chage the data chaged the free coeffcet b more tha 30%, the x coeffcet b about 3%, ad the coeffet of x b ol about %. If we repeat the calculato for the complete quadratc model we fd ŷ = 0.935x x, whch s about % of the lear term, ad about 3% dfferet the quadratc term Nolear Regresso he advatage of havg a respose surface whch s a lear fucto of the coeffcets, s that the regresso process requres ol the soluto of a sstem of lear equatos (the ormal equatos). However, ma stuatos such a respose surface wll ot ft the data well. I partcular, most of the work wth lear regresso s doe wth lear ad quadratc polomals, ad these are ver lmted terms of the kd of fuctos that the wll ft well. I olear regresso, we assume that the respose surface s the more geeral form f ( x) ( x, ), (3.3.) We aga have a vector of fucto values ( x ),,..., ad we tr to estmate b a vector b that mmzes the rms error erms r, (3.3.) where r s the resdual at the th data pot r ( x, b) (3.3.3) However, ulke the case of lear regresso, we caot perform the ft b solvg a set of lear equatos, ad stead we have to perform ucostraed mmzato, tpcall requrg a umercal soluto. he followg example demostrates the advatages of olear regresso. 44

18 Example 3.3. Gve the four data pots, () = 0, () = 7, (3) = 5, ad (4) = 4, fd a approprate fucto to ft ad compare to polomal ft. he large value of () dcates that the fucto ma become ubouded ear that value, so that a approxmato usg ratoal fuctos ma be approprate. So we tr b b, (3.3.4) b3 x he resduals at the four data pots are b b r b 0, r b 7, (3.3.5) b b 3 b b r 3 b 5, r 4 b 4, (3.3.6) b3 3 b3 4 Optmzg wth Mcrosoft Excel the sum of the four resduals elded the followg soluto b, b 7, 0. 6 b3 that s 7, (3.3.7) x 0.6 A comparable polomal approxmato wth 3 coeffcets wll be a quadratc polomal, ad performg regresso elds x 3x, (3.3.8) wth a estmated 5, a rather substatal error. he two approxmatos are compared the Fgure 3.5. Whle we caot use lear regresso for the fttg, we ca use t after the fttg s doe to obta a estmate of the errors the coeffcets. hs ca be doe b learzato about the soluto of the olear regresso. We learze Eq about the soluto b* obtaed from the least square ft r ( x, b*) x, b * bj, (3.3.9) j bj hat s, f we ft the resduals r usg as bass fuctos the dervatves b j, we ca get estmates of the stadard devato of the coeffcets. 3 45

19 Example 3.3. We learze the prevous example about the value of the coeffcets obtaed from the olear regresso, obtag b 7 r b b3. (3.3.0) x 0.6 x 0.6 A lear regresso aalss wth Mcrosoft Excel gves b , b , b Note that these cremets should be subtracted from the values we obtaed the prevous example for more accurate ft, because the are a ft to the resdual ŷ. From the lear regresso we get a estmate for the stadard error ˆ 0.0, ad stadard errors for the coeffcets as se( b ) 0.0, se( b ) 0.50, se ( b 3 ) hese dcate hgh cofdece the values of the coeffcets, sce the stadard errors are less tha 0% of the values of the coeffcet here are ma stadard forms of olear regresso that we wll cosder subsequet chapters cludg Krgg, eural etworks, ad support vector regresso. Fall, the process of fdg phscal costats from laborator or feld measuremets ca be vewed as a process of olear regresso, sce the depedece of the measuremets s rarel lear these costats. Ufortuatel, the problem of mmzg the rms errors for fttg phscal expermets ofte has a large umber of local optma, so that choosg the best aswer s ot alwas smple. 46

20 3.4 Oulers Both phscal expermets ad computer smulatos occasoall we have data wth large errors. I computer smulatos these ma reflect falures of the soluto algorthm, software mplemetato, or mstakes b the user of the software. Data pots wth large errors are called outlers. It s mportat to detect them ad ether remove or repar them, because the ca have a large detrmetal effect o the accurac of the respose surface. A stadard tool for detectg outlers s Iteratvel Reweghted Least Squares (IRLS) proce dures. I order to uderstad ther bass, let us cosder frst the weghted least square (WLS) procedure. Weghted least squares procedures mmze a weghted sum of the squares of the resduals. here are ma possble reasos for usg WLS rather tha stadard least squares. We ma have more cofdece some data tha others. We ma wat to weght more heavl pots that are close to the rego where we wll eed to predct the respose tha pots far awa. he ose at some pots ma be kow to be hgher tha other pots. I these cases, stead of usg Eq. 3.. we use ewrms we e We, (3.4.) where, w s the weght assocated wth the th pot, ad W s a dagoal matrx wth the weghts o the dagoal. Mmzg the weghted rms error, ewrms elds a modfed set of ormal equatos X WXb X W, (3.4.) Iteratvel reweghted least square procedures weght pots wth large resduals wth small weghts, wth the weght decreasg wth creasg magtude of the resdual. he the WLS procedure s performed. If the pot s a outler, the respose surface wll move awa from t, so ts resdual wll crease. We wll assg a smaller weght to the pot ad repeat the procedure. Evetuall, outlers are lkel to ed up wth zero or low weght. here are several weghtg schemes, see e.g., Mers ad Motgomer, p. 67. Oe of the smplest s Huber's w f e / ˆ (3.4.3) ˆ / e otherwse Example 3.4. We eed to estmate Youg's modulus E o the bass of four stress-stra measuremets. For values of stras of,, 3, ad 4, mllstras, we measures stress values of 9,, 36, ad 39 ks. Deotg the stress b ad the stra b x we perform a stadard least square ft =Ex. X, (3.4.4)

21 So we have X X 30, X 37, E 0.567Ms (3.4.5) e.567,0.867,4.3,3.67 gvg he vector of resduals s ˆ ee33.846, see ( ) , (3.4.6) he ol resdual greater tha ˆ s the thrd oe, so ts weght w We ow perform a weghted least squares ft, gvg us X WX 7.876, X W 95.5, E 0.46, (3.4.7) he ew vector of resduals se.457,.085,4.68,.830, gvg ˆ e We Aga, the ol resdual larger tha ˆ s the thrd oe, ad ts weght ow reduces to w Oe more terato gves us 3 X WX 6.904, X W , E 0.40, (3.4.8) We ca cotue teratg, or we ca be satsfed that the process has detfed the thrd pot as a outler ad dscard t. If we do that we get E = 9.95 Ms, wth a stadard error se(e) = Exercses. For Example 3.. calculate the rms error from the cross-valdato PRESS procedure ad compare to other estmates of the error.. Check the accurac of the quadratc respose surface Example 3.. the rego.5 D 3.5, 3 H 7. Fd the maxmum error, average error ad rms error (a) for the 9 data pots, (b) for the etre rego (usg the aaltcal expresso). For part (b) ou ma perform the calculato aaltcall, or ou ma cover the doma wth a grd of (0X0) pots ad calculate the error each oe of the 400 pots. (c) Calculate the error usg the PRESS procedure ad compare to the result of part (b) 3. Usg the data Example 3.. ft a quadratc polomal to the proft per ouce, ad perform the same error aalss as requested Problem. 4. Usg the data of Problem, use backward elmato to fd a complete quadratc polomal wth the hghest R. a. he check for the accurac of the ft compared to the a aaltcal proft per ca over the etre rego, ad compare to the results obtaed Problem. 48