Module III: Partial differential equations and optimization

Transcription

1 Module III: Partial differential equations and optimization Martin Berggren Department of Information Technology Uppsala University Optimization for differential equations Content Martin Berggren (UU) Opt. for DE / 41 Introduction Introduction, generalities Sensitivity analysis (in finite dimensions) Optimizing forcing terms/boundary conditions for elliptic PDE s (inverse or control problems) Optimizing coefficients: Material and topology optimization Optimizing geometry: Shape optimization Martin Berggren (UU) Opt. for DE / 41

2 Introduction What is the use of optimization for PDEs? Weather forecasting: Weather models (PDEs) needs initial conditions at each spatial point Only available data: limited set of local measurements at different times Find through optimization the initial condition that best matches the given observations ( 4D var ) Parameter estimation (e. g. material properties), nondestructive evaluation Optimizing geometrical properties: shapes and topologies Martin Berggren (UU) Opt. for DE / 41 Application examples Introduction Example I: Redesign of the ONERA M6 wing Reduce drag while keeping the lift and pitch moment constant. (PDE: the Euler equations of gas dynamics) Computations by Olivier Amoignon (2005) Initial wing (pressure) Initial (gray) and optimized (yellow) wing design Pressure on optimized wing Martin Berggren (UU) Opt. for DE / 41

3 Introduction Example II: Topology optimization (Borrvall & Petersson Linköping, 2001) Objective: Cutting out (say) 50 % of the material in a way that maximizes the stiffness of the remaining structure Martin Berggren (UU) Opt. for DE / 41 Problem structure Introduction Module I viewed objective functions or constraints as direct functions of the design variables (decision variables, control variables, parameters): Here: φ J φ u J design var. state objective, constraint The discrete design space may be of small or large dimension The discrete state space typically large (can have millions of degrees of freedom) The number of objectives, constraints typically small Often more intermediate steps Martin Berggren (UU) Opt. for DE / 41

4 Introduction Linear-algebra-type example State equation: Au = Bφ Objective function: j = c T u + ɛ 2 φ 2 φ U R m, u R n, n large; A: n-by-n, B: n-by-m A. Non-nested optimization formulation Viewing state equation as constraint min φ,u j (u,φ) subject to Au = φ, φ U Martin Berggren (UU) Opt. for DE / 41 Introduction Linear-algebra-type example, cont. Now define J(φ) = j (u,φ)where Au = φ B. Nested optimization formulation Viewing state equation only as intermediate step min φ J(φ) subject to φ U Martin Berggren (UU) Opt. for DE / 41

5 Introduction Nested and non-nested formulations The formulations suggest different algorithms Nested: Only φ is the optimization decision variables. States u will be kept feasible at each iteration. ( NAND : Nested Analysis and Design) Non-nested: Both φ and u are optimization decision variables. States will be feasible only at convergence. ( SAND : Simultaneous Analysis and Design) Martin Berggren (UU) Opt. for DE / 41 SAND vs. NAND Introduction SAND-type algorithms solves the state (and adjoint, se below) equations simultaneously with the optimization problem. Potentially very fast (goal: a cost corresponding to a fixed small multiple of state solves). Algorithms need to be run to full convergence, otherwise non-meaningful results (non-feasible states) NAND-type algorithms more standard. Can stop iterations when good enough. Can be very costly for expensive state equations. This module: only NAND-type algorithms. Martin Berggren (UU) Opt. for DE / 41

6 Directional derivatives The optimization algorithms from Module I need derivatives of the objective function and the constraints. Convenient to use directional derivatives (evaluations of the differential in an arbitrary direction) in derivations f = f(á +±Á) { f(á) ±f(á; ±Á) Á ±Á Black graph: the function f, Red graph: the differential δ f (φ) δ f (φ, δφ): the evaluation of the differential in direction δφ Martin Berggren (UU) Opt. for DE / 41 Directional derivatives f : U R objective function or constraint U a convex subset of R n or a function space (e. g. bounded or square-integrable functions) A design variation: δφ = φ φ; φ, φ U If φ U, by convexity, φ(s) = φ + s δφ stays in U for s [0, 1] f (φ + s δφ) f (φ) δ f (φ; δφ) = δ f (φ), δφ lim s 0 + s f (φ) δφ for U R, φ (if differentiable) = f (φ) T δφ for U R n, Df(φ) δφ for U L 2 ( ) Martin Berggren (UU) Opt. for DE / 41

7 Directional derivatives Analogous definition for functions with values in R n or any other normed space V (u : U V ) δu(φ; δφ) = lim s 0 + u(φ + s δφ) u(φ) s where the limit is in the sense of the norm on V, that is, δu(φ; δφ) is a directional derivative if lim u(φ + s δφ) u(φ) δu(φ; δφ) s 0 + s = 0 V Martin Berggren (UU) Opt. for DE / 41 Two ways to compute objective-function gradients State equation ( A square and nonsingular matrix): Au = φ Objective function: J(φ) = c T u Differentiate: A δu = δφ δ J = J T δφ = c T δu = c T ( A 1 δφ ) = ( A T c ) T δφ Martin Berggren (UU) Opt. for DE / 41

8 δ J = J T δφ = i J φ i δφ i = c T δu = c T ( A 1 δφ ) direct sensitivities = ( A T c ) T δφ adjoint equation Direct sensitivities: Compute each component of J by choosing successively δφ = all unit vectors. Computational complexity: The number of state-equation solves grows linearly with the number of design variables No extra state solves when changing objective function (i. e. c) Adjoint equations: Compute all components of J at once from A T c. Computational complexity Independent of the number of design variables Grows linearly with the number of objective functions Martin Berggren (UU) Opt. for DE / 41 Algorithms for gradient computations Many state equations are of the abstract form a(u) = b(φ) (However, often left-hand side also depend on φ: a(u,φ)= b(φ); treated in the exercises.) We consider several objective functions (say lift, drag,...) or constraints J i (φ) = f i (u(φ), φ), i = 1,...,m and want to compute J i, i = 1,...,m at some φ. Martin Berggren (UU) Opt. for DE / 41

9 Finite-differenced gradients Let e k = (0,...,0, 1, 0,...) T, with the non-zero component at position i J i J i (φ + ɛ e k ) J i (φ) = lim φ k ɛ 0 ɛ i = 1,...,m; k = 1,...,n 1. Solve a(u) = b(φ). 2. For k = 1,...,n do! Loop over design variables 2.1 Solve a(u k ) = b(φ + ɛ e k ) 2.2 For i = 1,...,m do! Loop over objective functions Set J i f i (u k,φ) f i (u,φ) φ k ɛ + f i(u,φ) φ k Martin Berggren (UU) Opt. for DE / 41 Easy to implement with existing, black-box software to solve the state equation. Computational effort: Essentially n state solves for each calculation of the gradient, that is, for each iteration of a gradient-based optimization algorithm. Note that the multiple objective functions do not cause any additional state solves. How to select ɛ? Too large ɛ yield an inaccurate gradient. Too small ɛ yields cancelation of significant digits. Can show that the optimal trade-off between accuracy and round off occurs for ɛ ɛ u, where ɛ u is unit round-off ( machine epsilon ) for the floating-point system (ɛ u = in IEEE double precision). Martin Berggren (UU) Opt. for DE / 41

10 The complex-variable trick A finite-difference technique that avoids cancelation effects. Let f = f (x) be a real-valued function of a single real variable. We want to approximate f (x). Let f = f (z) be the analytic continuation of f in a complex neighborhood of x. (Always exists if f is (real) analytic at x.) Martin Berggren (UU) Opt. for DE / 41 Using that dn f dz n = dn f dx n R on the real line, a Taylor-series expansion in imaginary direction at x R yields f (x + iɛ) = f (x) + iɛ f (x) + (iɛ)2 2 f (x) + (iɛ)3 6 f (x) + = f (x) ɛ2 2 f (x) + O(ɛ 4 ) + iɛ ( f (x) ɛ2 6 f (x) + O(ɛ 4 ) ) Thus, Re f (x + iɛ) = f (x) ɛ2 2 f (x) + O(ɛ 4 ) Im f (x + iɛ) ɛ = f (x) ɛ2 6 f (x) + O(ɛ 4 ) Martin Berggren (UU) Opt. for DE / 41

11 Hence f (x) Im f (x + iɛ) ɛ to second order in ɛ, and ɛ can be selected without concerns for cancelation! Procedure: 1. Solve a(u) = b(φ). 2. For k = 1,...,n do! Loop over design variables 2.1 Solve a(u k ) = b(φ + iɛ e k ) (in complex arithmetic) 2.2 For i = 1,...,m do! Loop over objective functions J i Set = Im f i (u k ) + f i(u,φ) φ k ɛ φ k, Martin Berggren (UU) Opt. for DE / 41 The derivative f can be obtained almost in full precision by choosing ɛ very small, say ɛ = Requires minor changes in the code: basically changing from real to complex arithmetic. Most operations, functions in a computer program posses analytic extensions. Watch out for: Absolute value: change x to x 2 (complex absolute value is not the analytic extensions of the real absolute value) Conditionals, max, min: operate on real part (differentiable and analytic if not exactly on the switch ) Martin Berggren (UU) Opt. for DE / 41

12 Same computational complexity as for finite-differences + the increased cost of using complex arithmetic Convenient in languages with build-in complex arithmetic such as Fortran or Matlab Less convenient and efficient for languages such as C++ lacking complex-arithmetic support In Matlab: watch out for the transpose operation on vectors: v means v H = v T. To obtain v T for a complex v, use v. Great for checking a code that computes exact gradients (by state sensitivities or the adjoint method) Martin Berggren (UU) Opt. for DE / 41 State sensitivities Utilizes the chain rule and computes explicitly the sensitivity of the states with respect to design changes. Linearize the state equation a(u) = b(φ) with respect to a design variation, A(u)δu = B(φ) δφ, where A ij = a i, B ij = b i, (Jacobian matrices) u j φ j Differentiation the objective functions yields where u f T i = δ J i (φ) = J i (φ) T δφ = δ f i (u(φ), φ) = u f i (u,φ) T δu + φ f i (u,φ) T δφ ( fi, f i,..., f ) i, φ fi T = u 1 u 2 u N ( fi, f i,..., f ) i φ 1 φ 2 φ n Martin Berggren (UU) Opt. for DE / 41

13 Procedure: 1. Solve a(u) = b(φ) 2. For k = 1,...,n do! Loop over design variables 2.1 Solve Au k = b (linearized state equation) φ k 2.2 For i = 1,...,m do! Loop over objective functions J i Set = u f i (u,φ) T u k + f i φ k φ k Martin Berggren (UU) Opt. for DE / 41 No parameter to choose: yields the exact gradient. Not easily implemented with black-box software: needs coding of the linearized state equations. If the code uses Newton s method to solve the nonlinear state equation, the Jacobian A(u) is already there (but probably not B(φ)) Computational complexity the same as for finite differences and the complex-variable trick. Martin Berggren (UU) Opt. for DE / 41

14 The adjoint-equation approach Recall the differentiated objective functions (i = 1,...,m) δ J i (φ) = J i (φ) T δφ = u f i (u,φ) T δu + φ f i (u,φ) T δφ (1) and the differentiated state equation (a(u) = b(φ)) A(u)δu = B(φ) δφ, (2) Multiply equation (2) with an arbitrary vector p T 0 = p T A δu p T B δφ = δu T A T p δφ T B T p. (3) Letting p i, i = 1,...,m, satisfy the adjoint equations A T p i = u f i (u,φ), equation (3) yields that δu T u f i (u,φ)= δφ T B(φ) T p i. (4) Martin Berggren (UU) Opt. for DE / 41 Substituting expression (4) into (1), we find δ J i (φ) = J i (φ) T δφ = ( ) B(φ) T T p i δφ + φ f i (u,φ) T δφ, and we may identify the gradient: J i (φ) = B(φ) T p i + φ f i (u,φ) Martin Berggren (UU) Opt. for DE / 41

15 Adjoint variables = Lagrange multipliers Consider U = R m and the optimization problem (non-nested form) min φ,u f (u,φ)subject to a(u) = b(φ) (P) Define the Lagrangian L (φ, u; p) = f (u,φ) p T( a(u) b(φ) ) Form Module I: the first-order necessary conditions for optimality is L (φ, u; p) = 0, or, δφ T φ L (φ, u; p) + δu T u L (φ, u; p) + δp T p L (φ, u; p) = 0 for each δφ R m, δu,δp R n Martin Berggren (UU) Opt. for DE / 41 L (φ, u; p) = f (u,φ) p T( a(u) b(φ) ) Differentiate with respect to each variable δp T p L (φ, u; p) = δp T( a(u) b(u) ) = 0 δu T u L (φ, u; p) = δu T( u f (u,φ) A(u) T p ) = 0 δφ T φ L (φ, u; p) = δφ T( φ f (u,φ)+ B(φ) T p ) = 0 [State equation] [Adjoint equation] [Gradient expression] Often called optimality system in the present context Martin Berggren (UU) Opt. for DE / 41

16 Gradient computations with adjoints. Procedure: 1. Solve a(u) = b(φ) (state equation) For i = 1,...,m do! Loop over objective functions 1.1 Solve A T p i = u f i, (adjoint equation) 1.2 Set J i (φ) = B(φ) T p i + φ f i (u,φ) Martin Berggren (UU) Opt. for DE / 41 Computational work: one state solve and m adjoint solves for each gradient evaluation. The computational work is independent of n, the size of the design space! Best computational efficiency when there are few objective functions (or constraints) but many design variables. Yields the exact gradient, no parameter. Not easily implemented with black-box software. Major coding effort to implement the adjoint equations: generally much more difficult to implement than linearized state equations. Martin Berggren (UU) Opt. for DE / 41

17 Automatic (or Algorithmic) Differentiation (AD) Both the state sensitivity and the adjoint method requires coding efforts AD requires access to source code but no coding, in principle Observation: Each line in a computer program is easy to differentiate Differentiation rules (e. g. product rule, chain rule) are completely mechanical Let the computer analyze each row in the program and calculate the derivative simultaneously as the function Martin Berggren (UU) Opt. for DE / 41 Assume we have a computer program computes the function f : R n R AD software turns this program into another program returning f (x) and f (x) Two ways to implement AD: Source transformation (compiler technology) Operator overloading Martin Berggren (UU) Opt. for DE / 41

18 AD with operator overloading Convenient in languages such as C ++ and Java Redefines real variables and redefines arithmetic operations to include derivative information u,v,w: real variables (their values depend on input vector x) α, β: constants (do not depend on x) For u,v, replace real data structure with an abstract data type also containing directional derivatives du, dv: u = ( ) du, v = u ( ) dv, w = v ( ) dw w For simplicity, assume scalar du, dv, dw (input vector x scalar) Straightforward to extend to vector du, dv, dw Martin Berggren (UU) Opt. for DE / 41 Examples Operation: v = αu. In code: v = α u. Operation redefined as v = ( ) α du αu Operation: w = uv. In code: w = u v. Operation redefined as w = ( ) udv + v du uv Operation: w = u. In code: w = u/v. Operation redefined as v du w = v 1 v dv 2 u v Martin Berggren (UU) Opt. for DE / 41

19 Change data type for each variable that may depend on x in program calculating f Providing the input ( ) ( dx x = 1x0 ) ( ) ( ) yields the output df f = f (x 0 ) f (x 0 ) Yields exact derivatives (up to machine precision) Computational effort grows linearly with the dimension of x, similarly as finite-differences Martin Berggren (UU) Opt. for DE / 41 Similar to the complex-variable trick, where the real part contained the function value and the imaginary part the derivative However, AD with operator overloading yields the whole gradient vector, not just one component (when using vector du) Less floating-point operations than the complex-variable trick Example, multiplication w = uv: AD: ( ) dw = w Complex variables: ( ) udv + v du uv w = uv = (u r + iu i )(v r + iv i ) = u r v r u i v }{{} i +i(u r v i + v r u i ) unnecessary (5) AD with operator overloading easy to use in C++ using FAD (see page 41) Martin Berggren (UU) Opt. for DE / 41

20 AD using source translation Is a compiler-like program. The AD tool associates to each scalar floating-point program variable v (also temporary ones) an n-vector dv. Each statement that assigns a value to a floating-point variable will be preceded by a statement that assigns, according to the chain rule, values to associated derivatives Example: the statement y = z sin(φ) will be replaced by the statements dy = z cos(φ) dφ + dz sin(dφ) y = z sin(φ) Martin Berggren (UU) Opt. for DE / 41 The compiler technology allows optimization of the produced code (as opposed to operator overloading) Above version of AD with source translation known as forward mode Computational complexity grows linearly with dimension of x AD in forward mode: essentially state sensitivities line-by-line in the code There is also an adjoint-version of AD: backward mode. Computational complexity then independent of dimension of x Backward mode may need excessive amounts of storage Martin Berggren (UU) Opt. for DE / 41

21 For AD software, see Examples (free): TAPENADE (former Odyssee). Source transformation; Fortran 77 and 95; forward and reverse (INRIA Sophia Antipolis, France) ADOL-C. Operator overloading; C/C++ (callable from Fortran); forward and reverse (Dresden Univ. of Techn. Germany) FAD. Simply a header file to be added to an existing C/C++ program to provide AD in forward mode by operator overloading (P. Aubert & N. Dicesare; can be downloaded from ADIFOR. Source transformation; Fortran 77; forward mode (Argonne Nat. Labs, Rice U., USA) ADIC. Source transformation; C/C++; forward mode (Argonne Nat. Labs) Martin Berggren (UU) Opt. for DE / 41