Originated from experimental optimization where measurements are very noisy Approximation can be actually more accurate than

Transcription

1 Surrogate (approxmatons) Orgnated from expermental optmzaton where measurements are ver nos Approxmaton can be actuall more accurate than data! Great nterest now n applng these technques to computer smulatons Computer smulatons are also subject to nose (numercal) However, smulatons are exactl repeatable, and f nose s small ma be vewed as exact. We wll dscuss polnomal response surfaces and Krgng to deal wth both stuatons

2 Polnomal response surface approxmatons Datasassumedtobe contamnated assumed to wth normall dstrbuted error of zero mean and standard devaton Response surface approxmaton has no bas error, and b havng more ponts than polnomal coeffcents t flters out some of the nose. Consequentl, approxmaton ma be more accurate than data

3 Fttng approxmaton to gven data Nos response model Data from n experments Lnear approxmaton Ratonal approxmaton Error measures e e rms av 1 n 1 n n 1 n 1 ˆ ( x, ) ˆ (xx, ) ˆ x ˆ ˆ( x ˆ( x, ) 1 x, ) e 1 max max x ˆ( x, )

4 Lnear Regresson Functonal form For lnear approxmaton Estmate of coeffcent vector denoted as b Rms error Mnmze rms error e T e=(-xb T ) T (-Xb T ) Dfferentate to obtan e e j rms X n ˆ ( x) x n b (x x ) e X T j 1 1 n e T e Xb X T j Beware of ll-condtonng!

5 Example Data: (0)=0, (1)=1, ()=1 ( 0) b 0 0 Ft lnear polnomal =b 0 +b 1 x (1) b0 b1 1 () b b 0 Then 0 3b 3b1 b 5b Obtan b 0 =1/3, b 1 =0.

6 Comparson wth alternate fts Errors for regresson ft e rms 0.47 e 0.44 e max av 0.67 To mnmze maxmum error obvousl =0.5. Then e av =e rms =e max =0.5 To mnmze average error, =0 e av =1/3, e max =1, e rms =0.577 rms What should be the order of the progresson from low to hgh?

7 Three lnes

8 Estmatng the accurac of the approxmaton (surrogate) From assumpton that error s due to normall dstrbuted uncorrelated random varables, get estmate to error standard devaton (called standard error) e T e n n Standard measure of accurac Coeffcent of multple determnaton measures how much of varablt n data s captured b approxmaton SS n 1 SS ˆ Adjusted coeffcent of multple determnaton accounts for the fttng bas R a 1 (1 R r n 1 n 11 ) n n ˆ R SS SS r

9 Cross valdaton Error estmates based on model assumptons are vulnerable For polnomal response surface approxmatons assumptons are rarel satsfed Cross valdaton dvdes data nto n g groups Ft the approxmaton to n g -1 groups, and use last group to estmate error. Repeat for each group When each group conssts of one pont, error called PRESS (predcton error sum of squares) Calculate error at each pont and then presentng r.m.s error e T 1 ep E X ( X X ) X Can be shown that 1 E Can be used onl f not ll-condtoned T

10 Estmatng error n coeffcents Dependng on desgn of experments, some coeffcents are more accuratel estmated than others Standard error n coeffcent s gven as se( b ) ˆ T X X 1 Coeffcents that are poorl estmated ma need to be dropped to mprove accurac of predctons usng approxmaton Unfortunatel, droppng one coeffcents changes t- statstcs for others Need to terate n droppng and addng coeffcents

11 Example 3..1 Gven data X Use Mcrosoft Excel to ft lnear and quadratc polnomals Compare standard errors and t-statstcs of coeffcents Y

12 Lnear ft SUMMARY OUTPUT Regresson Statstcs Multple R R Square Adjusted R Standard E Observato 5 ANOVA df SS MS F gnfcance F Regresson Resdual Total Coeffcentstandard Erro t Stat P-value Lower 95%Upper 95%Lower 95.0%Upper 95.0% Intercept X Varable RESIDUAL OUTPUT ObservatonPredcted Y Resduals

13 Quadratc ft SUMMARY OUTPUT Regresson Statstcs Multple R R Square Adjusted R Standard E Observato 5 ANOVA df SS MS F gnfcance F Regresson Resdual Total Coeffcentstandard Erro t Stat P-value Lower 95%Upper 95%Lower 95.0%Upper 95.0% Intercept X Varable X Varable RESIDUAL OUTPUT ObservatonPredcted t d YR Resduals

14 . Graphcal comparson

15 Exercse Generate data for the functon =x at n ponts on the nterval (-,) ) wth random nose unforml dstrbuted n (-0.5,0.5) Ft quadratc polnomal l for n>3 Check how man ponts ou need to get coeffcent of x wthn (0.9,1.1) 911)wth95% 95% confdence Compare standard error to PRESS error for each case Debug on fve pont data from Example 3..1

16 Desgn of experments Selectng samplng ponts s known as desgn of experments Classcal desgns nclude full-factoral desgn, nxnxnxn, central composte desgn,whch h s based on 3-level full-factoral desgn wth onl face ponts for the ntermedate level D-optmal desgn mnmzes varance- volume of coeffcent ellpsod Latn hpercube samplng (LHSDESIGN) dstrbutes ponts evenl through desgn space.

17 4..1 Interpolaton, extrapolaton and predcton varance Interpolaton s mathematcall contrasted to regresson or least-squares ft As mportant s the contrast between nterpolaton and extrapolaton Extrapolaton occurs when we are outsde the convex hull of the data ponts For hgh dmensonal spaces we must have extrapolaton!

18 Predcton varance Lnear regresson model Defne then Wth some algebra Standard error

19 Example 4..1 For a lnear polnomal RS =b 1 +b x 1 +b x fnd the predcton varance n the regon (a) For data at three vertces (b) For data at all four vertces

20 Data at three vertces Get ˆ Actual standard error multpled b nose ampltude At the vertces s ˆ Mnmum error At 1 1 s ˆ at x1 x 3 3 x 1 x 1, s 3ˆ

21 Data at four vertces Now And s 3 ˆ Error at vertces s 3 ˆ At the orgn mnmum s How can we reduce error wthout addng ponts? s 1 ˆ

22 Central Composte Desgn (CCD) Classcal DOE for quadratc RS FCCCD Move axal ponts to surface of the hpercube

23 Box-Benken desgns Dsturb onl a small number of varables from nomnal value. For these use all combnatons. For example for n=3, wth two varables changng

24 Varance optmal desgns A ke to most optmal DOE methods s moment matrx M=X T X/n The varance of the coeffcents and the predcton varance depend on (X T X )-1. A good desgn of experments wll mnmze the terms n ths matrx, especall the dagonal elements D-optmal desgns maxmze determnant of moment matrx Inversel proportonal to square of volume of confdence regon on coeffcents

25 Example Gven the model =b 1 x 1 +b x, and the two data ponts (0,0) and (1,0), fnd the optmum thrd data pont (p,q). We have 0 0 T 1 p pq T X 1 0 X X det( X X) q pq q p q So that the thrd pont s (p,1), for an value of p Fndng D-optmal desgn n hgher dmenson s dffcult optmzaton problems often solved heurstcall

26 Other crtera A-optmal mnmzes trace of nverse of moment matrx, mnmzes the sum of the varances of the coeffcents G-optmalt mnmzes the maxmum of the predcton varance.

27 Example For the prevous example, fnd the A-optmal desgn 0 0 Mnmum at (0,1) T 1 p pq T X 1 0 X X det( X X) q pq q p q T 1 1 q pq X X q pq 1 p T 1 1 p tr X X 1 q

28 Latn Hpercube Samplng (LHS) Unform Normal (Wss and Jorgensen) Advantages Space-fllng arbtrar number of desgn ponts Matlab LHSDESIGN can perform optmzaton to maxmze mnmum dstance between ponts

29 Revew of varous DOE Questons to ask Is nose an mportant ssue What surrogate do we prefer How man smulatons can we afford How man varables do we need to nclude Do we want to do all the smulatons at once or can we do adaptve samplng

30 Recommendatons Low-dmensonal spaces wth much nose Full factoral or ccd for box domans D-optmal desgns for rregular domans and wth adaptve samplng wth adaptve samplng Low-dmensonal spaces wthout much nose Mnmum bas, LHS and orthogonal arrras for box-lke domans Monte Carlo and optmzed dstance desgns for rregular domans

31 Recommendatons Hgh-dmensonal spaces wth nose Box-Benken and partal factoral or small CCD for regular shaped domans D-optmal desgns for rregularl shaped domans Hgh-dmensonal spaces wthout much nose LHS and optmzed dstance desgns

32 Surrogate modelng There are man surrogate models Polnomal response surface approxmaton (PRS or RSA) Krgng (KRG) Radal bass functons (RBF) Support vector regresson (SVR) etc. Major steps n surrogate modelng Choce of samplng data ponts (DOE) Smulatons Model constructon Model apprasal (error estmaton) Desgn of experments (DOE) Run numercal smulatons at DOE sampled ponts Construct surrogate models Error analss

33 Krgng N v ˆ( x ) ( x ) Z ( x ) 1 Lnear trend model Sstematc departure N v ( x ), (), s θ exp ( ) C Z x Z s θ x s 1 Named after a South Afrcan mnng engneer D. G. Krge Assumpton: Sstematc departures Z(x) are correlated Gaussan correlaton functon C(x,s,θ) s most popular Samplng data ponts Sstematc Departure Lnear Trend Model Krgng x

34 Radal bass neural networks ˆ( x ) N RBF 1 1 wa ( x ) w x ; Input a radbas b radbas n e n Neurons (Radal bass functons) at some of data ponts used to approxmate response. Other ponts used to estmate error W 1 a 1 User-defned constants: Spread constant: radus of nfluence x W a 1/b Error goal: mnmum sum of square of errors b Radal bass functon W 3 a ŷ(x) Input Output t Radal bass functons 0

35 Support vector regresson We do not care about error below epslon, but we We do not care about error below epslon, but we care about magntude of polnomal coeffcents

36 Suggested Exercses Exercse on page 16. Generate D-optmal and LHS desgns wth Matlab for a square and compare for dfferent number of ponts Source: Page11.htm