18-660: Numerical Methods for Engineering Design and Optimization

Transcription

1 8-66: Numerical Methods or Engineering Design and Optimization Xin Li Department o ECE Carnegie Mellon University Pittsburgh, PA 53 Slide

2 Overview Linear Regression Ordinary least-squares regression Minima optimization Design o eperiments Slide

3 Linear Regression Linear regression (also reerred to as response surace modeling) is widely used or many engineering problems We do not know the analytical orm o () But we can generate a set o sampling points or () Fit an approimate unction or () rom these sampling points () ( ) α b ( ) + α b ( ) + Model coeicients Basis unctions () is approimated as the linear combination o multiple basis unctions Slide 3

4 Linear Regression Major steps o linear regression Select a model template (e.g., polynomial unction) Generate a number o sampling points Compute perormance values at these sampling points Create a set o linear equations to solve model coeicients A simple eample () ep(), [-, ] We will use this simple eample to show how we can generally build a regression model rom sampling data Slide 4

5 Linear Regression Eample Step : select a model template ( ) b c + Step : generate a number o sampling points Samples Step 3: compute perormance values at these sampling points Samples () Slide 5

6 Linear Regression Eample Step 4: create linear equations or model coeicients ( ) b c + Samples () b c i-th sampling point values () values Slide 6

7 Linear Regression Eample Step 5: solve over-determined linear equations # o equations is greater than # o coeicients over-determined No eact solution eists to satisy all equations, but we can ind the least-squares solution: A α B min α Aα B Ordinary least-squares (OLS) regression Vector For a vector ε R M, ε is deined as: ε M ε i i Slide 7

8 Slide 8 Linear Regression Eample M B A ε ε ε ε α 3 i-th row Error at the i-th sampling point min B A α α ( ) M i i min α ε α A B α

9 Slide 9 Linear Regression Eample There are several possible ways to solve over-determined linear equations or linear regression We will eplain these algorithms in detail in uture lectures For now, you can simply use α A\B in MATLAB B A α M samples N coeicients (M > N)

10 Linear Regression Eample Step 5: solve over-determined linear equations.5.5 b c ep() Linear ep() b.486 c Linear model results in large error Slide

11 Quadratic Model Eample What i we build a quadratic model or y ep()? Select a model template ( ) a + b c + Generate a number o sampling points Samples Compute perormance values at these sampling points Samples () Slide

12 Quadratic Model Eample Create a set o linear equations to solve model coeicients ( ) a + b c + Samples () a b c () values Slide

13 Quadratic Model Eample Build quadratic model or y ep() a b c ep() Quadratic ep().5 a b c Quadratic model results in much better accuracy in this eample Slide 3

14 Linear Model vs. Quadratic Model Linear RSM Quadratic RSM ep ( ) ( ) ep + Regression model is dierent rom direct Taylor epansion E.g., dierent constant terms in linear and quadratic models they are selected to minimize the least-squares error 3.5 Linear ep() ep().5.5 Minimize leastsquares error Direct Taylor epansion Linear model or ep() Slide 4

15 Minima Optimization We can also solve over-determined linear equations to satisy other optimality criteria (i.e., not ordinary least-squares) A α B min α ma i i-th row o A A ( i,: ) α Bi Minimize the maimal absolute error i-th row A α B Error at the i-th sampling point Slide 5

16 Minima Optimization Other optimality criteria can be similarly ormulated A α B i-th row o A min α ma i A ( i,: ) α B B Minimize the maimal relative error i i i-th row A α B Error at the i-th sampling point These ormulations are minima optimization problems Slide 6

17 Minima Optimization General minima problems are diicult to solve Cost unction does not have continuous derivative ε(α) ma(ε, ε ) ε (α) ε (α) α Slide 7

18 Minima Optimization However, our minima problem or regression modeling can be re-ormulated into a special orm Consider the eample o absolute error minimization min α ma i A ( i,: ) α Bi Introduce a slack variable t min α, t S.T. Subject to t A A A (,: ) (,: ) ( M,:) α B α B α B M t t t Cost unction Constraints Slide 8

19 Minima Optimization min α, t S.T. t A A A (,: ) (,: ) ( M,:) α B α B α B M t t t min α, t S.T. t t A t A t A (,: ) (,: ) ( M,:) α B α B α B M t t t Re-written as a linear programming (LP) problem Both cost unction and constraints are linear No closed-orm solution eists or LP Can be numerically solved by an eicient (i.e., low compleity) and robust (i.e., global convergence) algorithm Slide 9

20 Design o Eperiments (DOE) We already know the basics or linear regression Open problem: How can we select ew samples to achieve good accuracy? A bad linear model eample: (, ) a + b + c - ( ) ( ) ( ) 3 Sampling points or linear model Slide

21 Slide Design o Eperiments (DOE) Linear model eample (continued) ( ) ( ) ( ) 3 ( ) c b a + +, 3 c b a Singular matri (cannot solve the coeicient b)

22 Design o Eperiments (DOE) Linear model eample (continued) - No variation is applied to - Add additional sampling points or Slide

23 Design o Eperiments (DOE) A bad quadratic model eample: (, ) a + a + a + b + b + c - ( ) ( ) ( 3 ) ( 4 ) ( ) 5 Sampling points or quadratic model Slide 3

24 Slide 4 Design o Eperiments (DOE) Quadratic model eample (continued) ( ) c b b a a a , c b b a a a Singular matri (cannot solve the coeicient a ) ( ) ( ) ( ) ( ) ( ) 5 4 3

25 Design o Eperiments (DOE) Quadratic model eample (continued) - Cross-product terms cannot be captured - Add additional sampling points or Slide 5

26 Design o Eperiments (DOE) Design o eperiments (DOE) is a research area that studies how to optimally select sampling points or modeling Given a model template (e.g., linear or quadratic unction), optimize sampling points or certain optimal criterion E.g., maimize modeling accuracy Numerical optimization may be required to ind the optimal sampling scheme D. Montgomery, Design and Analysis o Eperiments, John Wiley & Sons, 4 Slide 6

27 Summary Linear regression Ordinary least-squares regression Minima optimization Design o eperiments Slide 7