PDE-Constrained and Nonsmooth Optimization

Transcription

1 Frank E. Curtis October 1, 2009

2 Outline PDE-Constrained Optimization Introduction Newton s method Inexactness Results Summary and future work Nonsmooth Optimization Sequential quadratic programming (SQP) Gradient sampling (GS) SQP-GS Results Summary and future work Conclusion

3 Introduction Outline PDE-Constrained Optimization Introduction Newton s method Inexactness Results Summary and future work Nonsmooth Optimization Sequential quadratic programming (SQP) Gradient sampling (GS) SQP-GS Results Summary and future work Conclusion

4 Introduction PDE-constrained optimization min f (x) s.t. c E(x) = 0 c I(x) 0

5 Introduction PDE-constrained optimization min f (x) s.t. c E(x) = 0 c I(x) 0 (PDE) Problem is infinite-dimensional

6 Introduction Inverse problems Recover a parameter k(x) based on data collected from propagating waves

7 Introduction Inverse problems Recover a parameter k(x) based on data collected from propagating waves 1 X X min (y j (x m) y j,m ) 2 + α(β k 2 y,k 2 L 2 (Ω) + k 2 (L 2 (Ω)) n ) j m 8 y >< j (x) + S(x)k(x) 2 y j (x) S(x)(k 2 0 k(x)2 )y i j = 0, x in Ω s.t. y j = 0, x on Ω >: l(x) k(x) u(x), x in Ω

8 Introduction Inverse problems Recover a parameter k(x) based on data collected from propagating waves 1 X X min (y j (x m) y j,m ) 2 + α(β k 2 y,k 2 L 2 (Ω) + k 2 (L 2 (Ω)) n ) j m 8 y >< j (x) + S(x)k(x) 2 y j (x) S(x)(k 2 0 k(x)2 )y i j = 0, x in Ω s.t. y j = 0, x on Ω >: l(x) k(x) u(x), x in Ω

9 Introduction Optimal design Regional hyperthermia is a cancer therapy that aims at heating large and deeply seated tumors by means of radio wave adsorption Results in the killing of tumor cells and makes them more susceptible to other accompanying therapies; e.g., chemotherapy

10 Introduction Optimal design Computer modeling can be used to help plan the therapy for each patient, and it opens the door for numerical optimization The goal is to heat the tumor to a target temperature of 43 C while minimizing damage to nearby cells

11 Introduction Parameter estimation Weather forecasting If the initial state of the atmosphere (temperatures, pressures, wind patterns, humidities) were known at a certain point in time, then an accurate forecast could be obtained by integrating atmospheric model equations forward in time Flow described by Navier-Stokes and further sophistications of atmospheric physics and dynamics

12 Introduction Parameter estimation Limited amount of data (satellites, buoys, planes, ground-based sensors) Each observation is subject to error Nonuniformly distributed around the globe (satellite paths, densely-populated areas)

13 Newton s method Outline PDE-Constrained Optimization Introduction Newton s method Inexactness Results Summary and future work Nonsmooth Optimization Sequential quadratic programming (SQP) Gradient sampling (GS) SQP-GS Results Summary and future work Conclusion

14 Newton s method Nonlinear equations Newton s method F(x) = 0 F(x k )d k = F(x k ) Judge progress by the merit function φ(x) 1 2 F(x) 2 Direction is one of descent since φ(x k ) T d k = F(x k ) T F(x k )d k = F(x k ) 2 < 0 (Note the consistency between the step computation and merit function!)

15 Newton s method Equality constrained optimization Consider Lagrangian is min f (x) x R n s.t. c(x) = 0 L(x, λ) f (x) + λ T c(x) so the first-order optimality conditions are» f (x) + c(x)λ L(x, λ) = F(x, λ) = 0 c(x)

16 Newton s method Merit function Simply minimizing» ϕ(x, λ) = 1 F(x, 2 λ) 2 = 1 f (x) + c(x)λ 2 2 c(x) is generally inappropriate for constrained optimization We use the merit function φ(x; π) f (x) + π c(x) where π is a penalty parameter

17 Newton s method Minimizing a penalty function Consider the penalty function for min (x 1) 2, s.t. x = 0 i.e. φ(x; π) = (x 1) 2 + π x for different values of the penalty parameter π Figure: π = 1 Figure: π = 2

18 Newton s method Algorithm 0: Newton method for optimization (Assume the problem is sufficiently convex and regular) for k = 0, 1, 2,... Solve the primal-dual (Newton) equations»»» H(xk, λ k ) c(x k ) dk f (xk ) + c(x c(x k ) T = k )λ k 0 c(x k ) δ k Increase π, if necessary, so that Dφ k (d k ; π k ) 0 (e.g., π k λ k + δ k ) Backtrack from α k 1 to satisfy the Armijo condition φ(x k + α k d k ; π k ) φ(x k ; π k ) + ηα k Dφ k (d k ; π k ) Update iterate (x k+1, λ k+1 ) (x k, λ k ) + α k (d k, δ k )

19 Newton s method Convergence of Algorithm 0 Assumption The sequence {(x k, λ k )} is contained in a convex set Ω over which f, c, and their first derivatives are bounded and Lipschitz continuous. Also, (Regularity) c(x k ) T has full row rank with singular values bounded below by a positive constant (Convexity) u T H(x k, λ k )u µ u 2 for µ > 0 for all u R n satisfying u 0 and c(x k ) T u = 0 Theorem (Han (1977)) The sequence {(x k, λ k )} yields the limit» lim f (xk ) + c(x k )λ k k = 0 c(x k )

20 Inexactness Outline PDE-Constrained Optimization Introduction Newton s method Inexactness Results Summary and future work Nonsmooth Optimization Sequential quadratic programming (SQP) Gradient sampling (GS) SQP-GS Results Summary and future work Conclusion

21 Inexactness Large-scale primal-dual algorithms Computational issues: Large matrices to be stored Large matrices to be factored Algorithmic issues: The problem may be nonconvex The problem may be ill-conditioned Computational/Algorithmic issues: No matrix factorizations makes difficulties more difficult

22 Inexactness Nonlinear equations Compute F(x k )d k = F(x k ) + r k requiring (Dembo, Eisenstat, Steihaug (1982)) r k κ F(x k ), κ (0, 1) Progress judged by the merit function φ(x) 1 2 F(x) 2 Again, note the consistency... φ(x k ) T d k = F(x k ) T F(x k )d k = F(x k ) 2 +F(x k ) T r k (κ 1) F(x k ) 2 < 0

23 Inexactness Optimization Compute»»» H(xk, λ k ) c(x k ) dk f (xk ) + c(x c(x k ) T = k )λ k + 0 c(x k ) δ k» ρk r k satisfying» ρk r k» κ f (xk ) + c(x k )λ k, κ (0, 1) c(x k ) If κ is not sufficiently small (e.g., 10 3 vs ), then d k may be an ascent direction for our merit function; i.e., Dφ k (d k ; π k ) > 0 for all π k π k 1 Our work begins here... inexact Newton methods for optimization We cover the convex case, nonconvexity, irregularity, inequality constraints

24 Inexactness Model reductions Define the model of φ(x; π): m(d; π) f (x) + f (x) T d + π( c(x) + c(x) T d ) d k is acceptable if m(d k ; π k ) m(0; π k ) m(d k ; π k ) = f (x k ) T d k + π k ( c(x k ) c(x k ) + c(x k ) T d k ) 0 This ensures Dφ k (d k ; π k ) 0 (and more)

25 Inexactness Termination test 1 The search direction (d k, δ k ) is acceptable if»» ρk κ f (xk ) + c(x k )λ k, κ (0, 1) c(x k ) r k and if for π k = π k 1 and some σ (0, 1) we have m(d k ; π k ) max{ 1 2 d T k H(x k, λ k )d k, 0} + σπ k max{ c(x k ), r k c(x k ) } {z } 0 for any d

26 Inexactness Termination test 2 The search direction (d k, δ k ) is acceptable if ρ k β c(x k ), β > 0 and r k ɛ c(x k ), ɛ (0, 1) Increasing the penalty parameter π then yields m(d k ; π k ) max{ 1 2 d T k H(x k, λ k )d k, 0} + σπ k c(x k ) {z } 0 for any d

27 Inexactness Algorithm 1: Inexact Newton for optimization (Byrd, Curtis, Nocedal (2008)) for k = 0, 1, 2,... Iteratively solve»»» H(xk, λ k ) c(x k ) dk f (xk ) + c(x c(x k ) T = k )λ k 0 c(x k ) until termination test 1 or 2 is satisfied If only termination test 2 is satisfied, increase π so ( π k max π k 1, f (x k) T d k + max{ 1 d ) T 2 k H(x k, λ k )d k, 0} (1 τ)( c(x k ) r k ) Backtrack from α k 1 to satisfy δ k φ(x k + α k d k ; π k ) φ(x k ; π k ) ηα k m(d k ; π k ) Update iterate (x k+1, λ k+1 ) (x k, λ k ) + α k (d k, δ k )

28 Inexactness Convergence of Algorithm 1 Assumption The sequence {(x k, λ k )} is contained in a convex set Ω over which f, c, and their first derivatives are bounded and Lipschitz continuous. Also, (Regularity) c(x k ) T has full row rank with singular values bounded below by a positive constant (Convexity) u T H(x k, λ k )u µ u 2 for µ > 0 for all u R n satisfying u 0 and c(x k ) T u = 0 Theorem (Byrd, Curtis, Nocedal (2008)) The sequence {(x k, λ k )} yields the limit» lim f (xk ) + c(x k )λ k k = 0 c(x k )

29 Inexactness Handling nonconvexity and rank deficiency There are two assumptions we aim to drop: (Regularity) c(xk ) T has full row rank with singular values bounded below by a positive constant (Convexity) u T H(x k, λ k )u µ u 2 for µ > 0 for all u R n satisfying u 0 and c(x k ) T u = 0 e.g., the problem is not regular if it is infeasible, and it is not convex if there are maximizers and/or saddle points Without them, Algorithm 1 may stall or may not be well-defined

30 Inexactness No factorizations means no clue We might not store or factor» H(xk, λ k ) c(x k ) c(x k ) T 0 so we might not know if the problem is nonconvex or ill-conditioned Common practice is to perturb the matrix to be» H(xk, λ k ) + ξ 1I c(x k ) c(x k ) T ξ 2I where ξ 1 convexifies the model and ξ 2 regularizes the constraints Poor choices of ξ 1 and ξ 2 can have terrible consequences in the algorithm

31 Inexactness Our approach for global convergence Decompose the direction d k into a normal component (toward the constraints) and a tangential component (toward optimality) We impose a specific type of trust region constraint on the v k step in case the constraint Jacobian is (near) rank deficient

32 Inexactness Handling nonconvexity In computation of d k = v k + u k, convexify the Hessian as in» H(xk, λ k ) + ξ 1I c(x k ) c(x k ) T 0 by monitoring iterates Hessian modification strategy: Increase ξ 1 whenever u k 2 > ψ v k 2, ψ > ut k (H(x k, λ k ) + ξ 1I )u k < θ u k 2, θ > 0

33 Inexactness Algorithm 2: Inexact Newton (regularized) (Curtis, Nocedal, Wächter (2009)) for k = 0, 1, 2,... Approximately solve min 1 2 c(x k) + c(x k ) T v 2, s.t. v ω ( c(x k ))c(x k ) to compute v k satisfying Cauchy decrease Iteratively solve»»» H(xk, λ k ) + ξ 1 I c(x k ) dk f (xk ) + c(x c(x k ) T = k )λ k 0 δ k c(x k ) T v k until termination test 1 or 2 is satisfied, increasing ξ 1 as described If only termination test 2 is satisfied, increase π so ( π k max π k 1, f (x k) T d k + max{ 1 2 ut k (H(x ) k, λ k ) + ξ 1 I )u k, θ u k 2 } (1 τ)( c(x k ) c(x k ) + c(x k ) T d k ) Backtrack from α k 1 to satisfy φ(x k + α k d k ; π k ) φ(x k ; π k ) ηα k m(d k ; π k ) Update iterate (x k+1, λ k+1 ) (x k, λ k ) + α k (d k, δ k )

34 Inexactness Convergence of Algorithm 2 Assumption The sequence {(x k, λ k )} is contained in a convex set Ω over which f, c, and their first derivatives are bounded and Lipschitz continuous Theorem (Curtis, Nocedal, Wächter (2009)) If all limit points of { c(x k ) T } have full row rank, then the sequence {(x k, λ k )} yields the limit» lim f (xk ) + c(x k )λ k k = 0. c(x k ) Otherwise, and if {π k } is bounded, then lim ( c(x k))c(x k ) = 0 k lim f (x k) + c(x k )λ k = 0 k

35 Inexactness Handling inequalities Interior point methods are attractive for large applications Line-search interior point methods that enforce c(x k ) + c(x k ) T d k = 0 may fail to converge globally (Wächter, Biegler (2000)) Fortunately, the trust region subproblem we use to regularize the constraints also saves us from this type of failure!

36 Inexactness Algorithm 2 (Interior-point version) Apply Algorithm 2 to the logarithmic-barrier subproblem for µ 0 min f (x) µ qx ln s i, s.t. c E (x) = 0, c I (x) s = 0 i=1 Define H(x k, λ E,k, λ I,k ) 0 c E (x k ) c I (x k ) d x 3 k 6 0 µi 0 S k 4 c E (x k ) T 7 6 d s k δ E,k 5 c I (x k ) T S k 0 0 δ I,k so that the iterate update has»»» xk+1 xk d x + α k s k+1 s k k S k dk s Incorporate a fraction-to-the-boundary rule in the line search and a slack reset in the algorithm to maintain s max{0, c I (x)}

37 Inexactness Convergence of Algorithm 2 (interior-point) Assumption The sequence {(x k, λ E,k, λ I,k )} is contained in a convex set Ω over which f, c E, c I, and their first derivatives are bounded and Lipschitz continuous Theorem (Curtis, Schenk, Wächter (2009)) For a given µ, Algorithm 2 yields the same limits as in the equality constrained case If Algorithm 2 yields a sufficiently accurate solution to the barrier subproblem for each {µ j } 0 and if the linear independence constraint qualification (LICQ) holds at a limit point x of {x j }, then there exist Lagrange multipliers λ such that the first-order optimality conditions of the nonlinear program are satisfied

38 Results Outline PDE-Constrained Optimization Introduction Newton s method Inexactness Results Summary and future work Nonsmooth Optimization Sequential quadratic programming (SQP) Gradient sampling (GS) SQP-GS Results Summary and future work Conclusion

39 Results Implementation details Incorporated in IPOPT software package (Wächter) inexact algorithm yes Linear systems solved with PARDISO (Schenk) SQMR (Freund (1994)) Preconditioning in PARDISO incomplete multilevel factorization with inverse-based pivoting stabilized by symmetric-weighted matchings Optimality tolerance: 1e-8

40 Results CUTEr and COPS collections 745 problems written in AMPL 645 solved successfully 42 real failures Robustness between 87%-94% Original IPOPT: 93%

41 Results Helmholtz N n p q # iter CPU sec (per iter) (21.833) (149.66) (2729.1)

42 Results Boundary control min 1 2 R Ω (y(x) yt(x))2 dx s.t. (e y(x) y(x)) = 20 in Ω y(x) = u(x) on Ω 2.5 u(x) 3.5 on Ω where y t(x) = x 1 (x 1 1)x 2 (x 2 1) sin(2πx 3 ) N n p q # iter CPU sec (per iter) (0.2165) (7.9731) (380.88) Original IPOPT with N = 32 requires 238 seconds per iteration

43 Results Hyperthermia treatment planning min 1 R 2 Ω (y(x) yt(x))2 dx s.t. y(x) 10(y(x) 37) = u M(x)u in Ω 37.0 y(x) 37.5 on Ω 42.0 y(x) 44.0 in Ω 0 where u j = a j e iφ j, M jk (x) =< E j (x), E k (x) >, E j = sin(jx 1 x 2 x 3 π) N n p q # iter CPU sec (per iter) (0.3367) (59.920) Original IPOPT with N = 32 requires 408 seconds per iteration

44 Results Groundwater modeling min 1 R 2 Ω (y(x) yt(x))2 dx α R Ω [β(u(x) ut(x))2 + (u(x) u t(x)) 2 ]dx s.t. (e u(x) y i (x)) = q i (x) in Ω, i = 1,..., 6 y i (x) n = 0 on Ω Z y i (x)dx = 0, i = 1,..., 6 Ω 1 u(x) 2 in Ω where q i = 100 sin(2πx 1 ) sin(2πx 2 ) sin(2πx 3 ) N n p q # iter CPU sec (per iter) ( ) ( ) ( ) Original IPOPT with N = 32 requires approx. 20 hours for the first iteration

45 Summary and future work Outline PDE-Constrained Optimization Introduction Newton s method Inexactness Results Summary and future work Nonsmooth Optimization Sequential quadratic programming (SQP) Gradient sampling (GS) SQP-GS Results Summary and future work Conclusion

46 Summary and future work Summary We have a new framework for inexact Newton methods for optimization Convergence results are as good (and sometimes better) than exact methods Preliminary numerical results are encouraging

47 Summary and future work Future work Tune the method for specific applications Incorporate useful techniques such as filters, second-order corrections, specialized preconditioners Use (approximate) elimination techniques so that larger (e.g., time-dependent) problems can be solved

48 SQP Outline PDE-Constrained Optimization Introduction Newton s method Inexactness Results Summary and future work Nonsmooth Optimization Sequential quadratic programming (SQP) Gradient sampling (GS) SQP-GS Results Summary and future work Conclusion

49 SQP Constrained optimization of smooth functions Consider constrained optimization problems of the form min x f (x) s.t. c(x) 0 where f and c are smooth (equality constraints OK, too) At x k, solve the SLP/SQP subproblem min d f k + fk T d + 1 d T H 2 k d s.t. c k + c T k d 0, to compute the search direction d k d k

50 SQP SQP illustration: Objective model

51 SQP SQP illustration: Constraint model

52 SQP Practicalities Since the linearized constraints may be inconsistent, we solve min d ρ(f k + fk T d) + X s i + 1 d T H 2 k d s.t. c k + c T k d s, s 0, where ρ > 0 is a penalty parameter We perform a line search on the penalty function to promote global convergence φ(x; ρ) ρf (x) + X max{0, c i (x)}

53 SQP Line search A model of the penalty function is given by q k (d; ρ) ρ(f k + fk T d) + X max{0, ck i + ck i T d} + 1 d T H 2 k d Solving the SQP subproblem is equivalent to minimizing q k (d; ρ) The reduction in q k (d; ρ) yielded by d k is q k (d k ; ρ) q k (0; ρ) q k (d k ; ρ) We impose the sufficient decrease condition φ(x k + α k d k ; ρ) φ(x k ; ρ) ηα k q k (d k ; ρ)

54 SQP Penalty-SQP method for k = 0, 1, 2,... Solve the SQP subproblem min d or, equivalently, solve ρ(f k + fk T d) + X s i + 1 d T H 2 k d s.t. c k + c T k d s, s 0 min d q k (d; ρ) ρ(f k + fk T d) + X max{0, ck i + ck i T d} + 1 d T H 2 k d to compute d k Backtrack from α k = 1 to satisfy φ(x k + α k d k ; ρ) φ(x k ; ρ) ηα k q k (d k ; ρ) Update x k+1 x k + α k d k

55 GS Outline PDE-Constrained Optimization Introduction Newton s method Inexactness Results Summary and future work Nonsmooth Optimization Sequential quadratic programming (SQP) Gradient sampling (GS) SQP-GS Results Summary and future work Conclusion

56 GS Unconstrained optimization of nonsmooth functions Consider the unconstrained optimization problem min x f (x) where f may be nonsmooth (but is at least locally Lipschitz) The prototypical example is the absolute value function:

57 GS Clarke subdifferential Suppose f is differentiable over an open dense set D Let B(x, ɛ) {x x x ɛ} The Clarke subdifferential is f (x ) = \ cl conv f (B(x, ɛ) D) ɛ>0 A point x is called Clarke stationary if 0 f (x )

58 GS ɛ-stationarity The Clarke ɛ-subdifferential is given by f (x, ɛ) = cl conv f (B(x, ɛ) D) A point x is called ɛ-stationary if 0 f (x, ɛ)... find ɛ-stationary point, reduce ɛ, find ɛ-stationary point,...

59 GS Gradient sampling: Robust steepest descent (Burke, Lewis, Overton, 2005) We restrict iterates to the open dense set D Ideally, at x k, for a given ɛ we would solve min d f k + max x B k { f (x) T d} d T H k d where B k = B(x k, ɛ) D However, we can only approximate this step by solving min d f k + max x B k { f (x) T d} d T H k d where B k = {x k, x k1,..., x kp } B(x k, ɛ) D

60 GS GS illustration: Objective model

61 GS GS illustration: Objective model

62 GS GS illustration: Objective model

63 GS GS illustration: Objective model

64 GS GS illustration: Objective model

65 GS GS method for k = 0, 1, 2,... Sample points {x k1,..., x kp } in B(x k, ɛ) D Solve the GS subproblem min d f k + max x B k { f (x) T d} d T H k d to compute d k Backtrack from α k = 1 to satisfy f (x k + α k d k ) f (x k ) ηα k d k 2 Update x k+1 x k + α k d k (to ensure x k+1 D) If d k ɛ, then reduce ɛ

66 SQP-GS Outline PDE-Constrained Optimization Introduction Newton s method Inexactness Results Summary and future work Nonsmooth Optimization Sequential quadratic programming (SQP) Gradient sampling (GS) SQP-GS Results Summary and future work Conclusion

67 SQP-GS Constrained optimization of nonsmooth functions Consider constrained optimization problems of the form min x f (x) s.t. c(x) 0 where f and c may be nonsmooth (equality constraints OK, too) We may consider solving or even min x min x but this makes me... :-( φ(x; ρ) ρf (x) + X max{0, c i (x)} ϕ(x; ρ) ρf (x) + max max{0, c i (x)} i

68 SQP-GS SQP and GS The SQP subproblem is The GS subproblem is min d min d ρz + X s i d T H k d s.t. f k + fk T d z c k + c T k d s, s 0 z d T H k d s.t. f k + f (x) T d z, x B k

69 SQP-GS SQP-GS The SQP-GS subproblem is min ρz + X s i + 1 d T H d,z,s 2 k d s.t. f k + f (x) T d z, x B 0 k c i k + c i (x) T d s i, s i 0, x B i k, i = 1,..., m where B i k = {x k, x i k1,..., x i kp} B(x k, ɛ) for i = 0,..., m This is equivalent to min d ρ max(f k + f (x) T d)+ X max x B k 0 x B i k max{0, c i k + c i (x) T d, 0}+ 1 2 d T H k d i.e., min d q k (d; ρ), where now q k (d; ρ) is a robust model of φ(x; ρ)

70 SQP-GS SQP-GS illustration: Constraint model

71 SQP-GS SQP-GS illustration: Constraint model

72 SQP-GS SQP-GS illustration: Constraint model

73 SQP-GS SQP-GS illustration: Constraint model

74 SQP-GS SQP-GS illustration: Constraint model

75 SQP-GS SQP-GS method for k = 0, 1, 2,... Sample points {x i k1,..., x i kp} in B(x k, ɛ) D i for i = 0,..., m Solve the SQP-GS subproblem min ρz + X s i + 1 d T H d,z,s 2 k d s.t. f k + f (x) T d z, x B 0 k c i k + c i (x) T d s i, s i 0, x B i k, i = 1,..., m to compute d k Backtrack from α k = 1 to satisfy φ(x k + α k d k ; ρ) φ(x k ; ρ) ηα k q k (d k ; ρ) Update x k+1 x k + α k d k (to ensure x k+1 i D i ) If q k (d k ; ρ) ɛ, then reduce ɛ

76 SQP-GS Global convergence Assumption 1: The functions f and c i, i = 1,..., m, are locally Lipschitz and continuously differentiable on open dense subsets of R n Assumption 2: The sequence of iterates and sample points are contained in a convex set over which the functions f and c i, i = 1,..., m, and their first derivatives are bounded Assumption 3: For universal constants ξ ξ > 0, the Hessian matrices satisfy ξ d 2 d T H k d ξ d 2 for all d R n

77 SQP-GS Global convergence Lemma 1: q k (d k ; ρ) = 0 if and only if x k is ɛ-stationary Lemma 2: The one-sided directional derivative of the penalty function satisfies φ (d k ; ρ) d T k H k d k < 0 and so d k is a descent direction for φ(x; ρ) at x k Lemma 3: Suppose the sample size is p n + 1. If the current iterate x k is sufficiently close to a stationary point x of the penalty function φ(x; ρ), then there exists a nonempty open set of sample sets such that the solution to the SQP-GS subproblem d k yields an arbitrarily small q k (d k ; ρ) Carathéodory s Theorem Theorem: With probability one, every cluster point of {x k } is feasible and stationary for φ(x; ρ)

78 Results Outline PDE-Constrained Optimization Introduction Newton s method Inexactness Results Summary and future work Nonsmooth Optimization Sequential quadratic programming (SQP) Gradient sampling (GS) SQP-GS Results Summary and future work Conclusion

79 Results Implementation Prototype implementation in MATLAB (available soon?) QP subproblems solved with MOSEK BFGS approximations of Hessian of penalty function (Lewis and Overton, 2009) ρ decreased conservatively

80 Results Example 1: Nonsmooth Rosenbrock min x 8 x 2 1 x 2 + (1 x 1) 2 s.t. max{ 2x 1, 2x 2} 1

81 Results Example 1: Nonsmooth Rosenbrock

82 Results Example 2: Entropy minimization Find a N N matrix X that solves! KY min ln λ j (A X T X ) X j=1 s.t. X j = 1, j = 1,..., N where λ j (M) denotes the jth largest eigenvalue of M, A is a real symmetric N N matrix, denotes the Hadamard matrix product, and X j denotes the jth column of X

83 Results Example 2: Entropy minimization N n K f (SQP-GS) f (GS) e e e e e e e e e e e e e e e e-02

84 Results Example 3(a): Compressed sensing (l 1 norm) Recover a sparse signal by solving min x x 1 s.t. Ax = b where A is a submatrix of a discrete cosine transform (DCT) matrix

85 Results Example 3(a): Compressed sensing (l 1 norm)

86 Results Example 3(b): Compressed sensing (l 0.5 norm) Recover a sparse signal by solving min x x 0.5 s.t. Ax = b where A is a submatrix of a discrete cosine transform (DCT) matrix

87 Results Example 3(b): Compressed sensing (l 0.5 norm) Figure: k = 1

88 Results Example 3(b): Compressed sensing (l 0.5 norm) Figure: k = 10

89 Results Example 3(b): Compressed sensing (l 0.5 norm) Figure: k = 25

90 Results Example 3(b): Compressed sensing (l 0.5 norm) Figure: k = 50

91 Results Example 3(b): Compressed sensing (l 0.5 norm) Figure: k = 200

92 Results Example 3(b): Compressed sensing (l 0.5 norm)

93 Summary and future work Outline PDE-Constrained Optimization Introduction Newton s method Inexactness Results Summary and future work Nonsmooth Optimization Sequential quadratic programming (SQP) Gradient sampling (GS) SQP-GS Results Summary and future work Conclusion

94 Summary and future work Summary We have presented a globally convergent algorithm for the solution of constrained, nonsmooth, and nonconvex optimization problems The algorithm follows a penalty-sqp framework and uses Gradient Sampling to make the search direction calculation robust Preliminary results are encouraging

95 Summary and future work Future work Tune updates for ɛ and ρ Allow for special handling of smooth/convex/linear functions Investigate SLP vs. SQP Extensions for particular applications; e.g., specialized sampling

96 Thanks!!