18-660: Numerical Methods for Engineering Design and Optimization

Transcription

1 8-660: Numerica Methods for Engineering esign and Optimization in i epartment of ECE Carnegie Meon University Pittsburgh, PA 523 Side

2 Overview Conjugate Gradient Method (Part 4) Pre-conditioning Noninear conjugate gradient method Side 2

3 Conjugate Gradient Method Step : start from an initia guess (0), and set k 0 Step 2: cacuate Step 3: update soution ( 0) ( 0) ( 0) ( k + ) ( k ) ( k ) ( k ) ( k ) + µ where µ B A A Step 4: cacuate residua ( k +) ( k ) ( k ) ( k ) µ Step 5: determine search direction ( k + ) ( k + ) ( k ) k +, k A + β where β Step 6: set k k + and go to Step 3 k +, k ( k + ) T ( k + ) Side 3

4 Convergence ate ( ) k κ( A) κ( A) + k + 0 ( ) Conjugate gradient method has sow convergence if κ(a) is arge I.e., A B is i-conditioned In this case, we want to improve convergence rate by preconditioning Side 4

5 Pre-Conditioning Key idea Convert A B to another equivaent equation A B Sove à B by conjugate gradient method Important constraints to construct A B à is symmetric and positive definite so that we can sove it by conjugate gradient method à has a sma condition number so that we can achieve fast convergence Side 5

6 Pre-Conditioning A A B A A T is symmetric and positive definite, if A is symmetric and positive definite B T T A B B T T T ( A ) A T A T ( ) ( ) 0 T T A T > Side 6

7 Pre-Conditioning A A T has a sma condition number, if is propery seected T T A B B In theory, can be optimay found by Choesky decomposition T A T T T A I (Identify matrix) However, Choesky decomposition is not efficient for arge, sparse probems If we know Choesky decomposition, we amost sove the equation no need to use conjugate gradient method Side 7

8 Pre-Conditioning A In practice, can be constructed in many possibe ways T T A B B iagona pre-conditioning (or Jacobi pre-conditioning) Scae A aong coordinate axes a a 22 Side 8

9 Pre-Conditioning Incompete Choesky pre-conditioning A T T A B B is ower-trianguar Few or no fi-ins are aowed A T (not exacty equa) Side 9

10 Pre-Conditioning Step : start from an initia guess (0), and set k 0 Step 2: cacuate Step 3: update soution ( k ) T ( k ) ( k + ) ( k ) ( k ) ( k ) ( k ) + µ where µ A Step 4: cacuate residua 0 ( ) ( 0) T ( 0) B Step 5: determine search direction Step 6: set k k + and go to Step 3 µ ( k + ) ( k + ) ( k ) A ( k + ) ( k ) ( k ) T ( k ) A ( k ) T T ( k ) ( k + ) T ( k + ) + βk +, k where βk +, k Side 0

11 Pre-Conditioning A B ( 0 ) ( 0) T ( 0) B A ( k ) T ( k ) ( k + ) ( k ) ( k ) ( k ) ( k ) + µ where µ A ( k + ) ( k ) ( k ) T ( k ) µ A ( k + ) ( k + ) ( k ) shoud not be expicity computed A ( k ) T T ( k ) ( k + ) T ( k + ) + βk +, k where βk +, k Instead, Y W or Y T W (where W is a vector) shoud be computed by soving inear equation Y W or T Y W T T B Side

12 Side 2 Pre-Conditioning iagona pre-conditioning is a diagona matrix Y W or Y T W can be found by simpy scaing W Y a a a w y a w y

13 Pre-Conditioning Incompete Choesky pre-conditioning is ower-trianguar Y W or Y T W can be found by backward substitution Y W y 2 y w ( w y ) Side 3

14 Side 4 Pre-Conditioning Once is known, is cacuated as T Sove inear equation T by backward substitution B A T T A B ( ),, N N N N N N N NN N N x x x x x a a a x x a x x iagona pre-conditioning Incompete Choesky preconditioning

15 Noninear Conjugate Gradient Method Conjugate gradient method can be extended to genera (i.e., non-quadratic) unconstrained noninear optimization min 2 T A B T + C min f ( ) Noninear programming A B Quadratic programming A number of changes must be made to sove noninear optimization probems Side 5

16 Noninear Conjugate Gradient Method Step : start from an initia guess (0), and set k 0 Step 2: cacuate Step 3: update soution ( 0) ( 0) ( 0) ( k + ) ( k ) ( k ) ( k ) ( k ) + µ where µ B A A Step 4: cacuate residua ( k +) ( k ) ( k ) ( k ) µ Step 5: determine search direction ( k + ) ( k + ) ( k ) k +, k A + β where β Step 6: set k k + and go to Step 3 k +, k ( k + ) T ( k + ) Side 6

17 Noninear Conjugate Gradient Method New definition of residua ( 0) ( 0) ( 0) ( k +) ( k ) ( k ) ( k ) B A µ A Quadratic programming [ ] ( k ) ( k ) f Noninear programming esidua is defined by the gradient of f() If * is optima, f(*) 0 f(*) B A for quadratic programming Side 7

18 Noninear Conjugate Gradient Method New formua for conjugate search directions ( k + ) ( k + ) ( k ) + β where β k +, k Quadratic programming k +, k ( k + ) T ( k + ) Ideay, search directions shoud be computed by Gram- Schmidt conjugation of residues In practice, we often use approximate formuas β k +, k ( k + ) T ( k + ) Fetcher-eeves formua β k +, k [ ] ( k + ) T ( k + ) ( k ) Poak-ibiere formua Side 8

19 Noninear Conjugate Gradient Method Optima step size cacuated by one-dimensiona search ( k + ) ( k ) ( k ) ( k ) ( k ) + µ where µ Quadratic programming A µ (k) cannot be cacuated anayticay Optimize µ (k) by one-dimensiona search µ ( ) ( k +) ( k ) ( k ) ( k ) [ ] f + [ ] min f µ k Side 9

20 Noninear Conjugate Gradient Method Step : start from an initia guess (0), and set k 0 Step 2: cacuate Step 3: update soution µ ( ) Step 4: cacuate residua ( 0) ( 0) ( 0) Step 5: determine search direction (Fetcher-eeves formua) Step 6: set k k + and go to Step 3 ( k ) ( k ) ( k ) [ + ] min f µ k β k +, k ( k + ) T ( k + ) f [ ] ( k + ) ( k +) f ( k +) ( k ) ( k ) ( k ) + µ [ ] ( k + ) ( k + ) ( k ) + βk +, k Side 20

21 Noninear Conjugate Gradient Method Gradient method, conjugate gradient method and Newton method Conjugate gradient method is often preferred for many practica arge-scae engineering probems Gradient Conjugate Gradient Newton st-order erivative Yes Yes Yes 2nd-Order erivative No No Yes Pre-conditioning No Yes No Cost per Iteration ow ow High Convergence ate Sow Fast Fast Preferred Probem Size arge arge Sma Side 2

22 Summary Conjugate gradient method (Part 4) Pre-conditioning Noninear conjugate gradient method Side 22